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“The World Health Organization Quality of Life (WHOQOL) assessment with its 
comprehensive 24 parameters is an outstanding but concomitantly daunting tool to 
measure the quality of life in individuals. Without a valid and reliable technology- 
based approach to this assessment tool, it is a set of parameters without the fullest 
impact on health with its measurements and interventions. 

A technological and computational dyadic approach to the expansive WHOQOL 
tool would be the ideal visionary strategy. We should all applaud Sharon Wulfovich 
and Katarzyna Wac for their innovative philosophy espoused in the book Quantifying 
Quality of Life: Incorporating Daily Life into Medicine. This paradigm shift from a 
subjective, qualitative approach to a more objective, quantitative approach with 
human oversight renders this WHOQOL tool more accurate and efficient not to 
mention more continual and expedient. These aforementioned elements will be 
invaluable for the future of health in both individuals and populations. 

We are in the midst of a global health crisis and its apocalyptic consequences. 
The erudite author Arundhati Roy wrote, “the pandemic is a portal, a gateway 
between one world and the next.” This innovative approach to measuring the quality 
of health elucidated by Wulfovich and Wac is a very much welcomed part of this 
portfolio of tools we should bring along as we traverse through the portal into the 
new world.” 

—Anthony C. Chang, Chief Intelligence and Innovation Officer, 
Children’s Hospital of Orange County, CA, USA 


“Quantifying Quality of Life is the first book of its kind by describing how to 
quantify both the short-term and long-term determinants of health and wellbeing 
over an individual’s life course. With 5G broadband cellular networks on the horizon 
and the continual enthusiasm in the quantifying-self movement around the world, 
this book offers the methodologies and the tools for daily continuous assessment of 
the quality of life. 

As the field gains more experience on how to quantify the quality of life by 
leveraging both technology and patient-reported outcomes, we will begin to see 
more relationships emerging between different domains and variables being 
measured, e.g. exercise and mental health; social relations and productivity; mobility 
and self-esteem, etc. This will help us gain new insights for building a better 
environment and society for all citizens. 

This book is a remarkable first step towards understanding the science of an 
emerging and expanding field combining digital health, artificial intelligence, and 
Internet-of-Things. The editors and authors are to be congratulated for this 
pioneering work!” 

—C. Jason Wang, Director, Center for Policy, 

Outcomes, and Prevention, Associate Professor of Pediatrics, Medicine, 
and Health Research and Policy, Co-chair, Mobile Health 

and Other New Technologies, Center for Population Health Sciences, 
Stanford University, USA 
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“The arrival of smartphones and technology-enabled tools and methods has sparked 
a veritable flood of what is called “Real-World Monitoring (RWM)-" It allowed for 
creating unprecedented evidence data and patient-reported quality-of-life outcomes. 
Unlike traditional patient assessments via questionnaires or similar backward 
directed surveys, the new digital applications can enable objective, real-time data 
collection. This adds a completely new dimension to data quality and expressiveness. 
QoL Measurements and PRO’s are relevant to all healthcare stakeholders including 
patients. Doctors are able to adapt and individualize therapies more easily and can 
communicate to patients well outside of their practice and routine examinations. 
Payers can finally move away from a “fee-for-service” to an outcomes-based 
remuneration model—which is an important requirement to implement connected 
care structures in an increasingly complex and costly care environment. Life 
Sciences firms as well as the MedTech and Health IT industries are in a position to 
comply with new approval and certification requirements from authorities, such as 
the FDA or the EMA, to include PRO’s and not only traditional clinical data into 
their studies and applications. Finally, for regulators, solid and reliable outcomes 
data will be the basis to steer and manage healthcare systems much better than in the 
past—especially when it comes to placing innovation incentives and adapting 
processes or care structures. A multitude of health apps are available today and 
more of them are arriving each day which measure all sorts of personal health scores 
and figures. 

However, monitoring isolated values without relating them to each other is of 
little use. Therefore, it is important to distinguish patient-reported QoL data—and 
this also includes data coming from digital tools and applications (called Technology 
Reported Outcomes, TechROs). These tools only make sense if they are incorporated 
into a coherent body image to avoid biases, misinterpretation, or false therapeutic 
conclusions. This book not only looks at different elements that make up a holistic 
QoL picture, but it also demonstrates scientific methods to assess them which will 
allow them to be put into a meaningful clinical context. This book is a useful guide 
for all healthcare stakeholders.” 

—Rainer Herzog, Managing Director, 
Digital Health Partners, Germany 


“Recently gathered insights indicate that conditions such as heart disease, stroke, 
and diabetes are essentially the pathophysiologic endpoints of a complex interplay 
of an individual’s life experiences in five domains of influence: genetic predisposition, 
social circumstances, physical environment, behavior, and medical care. 

It now becomes possible to measure QoL objectively and quantitatively using 
validated methods and to incorporate these into both everyday clinical care and 
research to measure the effect of various treatments and strategies in the real world. 
We have witnessed an influx of market adaption for digital health and quality of life 
technologies this past year from telemedicine and virtual clinical trials to remote 
patient monitoring and quality of life assessments. Digital health technologies are 


here to stay and will play a critical role in the way we treat and cure diseases while 
empowering patients to manage their self-care. 

I was delighted to see the complex and rather challenging topic “Quantifying 
Quality of Life: Incorporating Daily Life into Medicine” being tackled by Prof 
Katarzyna Wac and Sharon Wulfovich. I found the QoL domains: physical health, 
psychological, social relationships, and environment to be meticulously addressed 
in this book with a state-of-the-art approach taken for the interpretation of the 24 
related variables.” 

—Mathieu Ghadanfar, President, 

M-Ghadanfar Consulting Life Sciences, 
Cardiovascular Physician | Healthcare Executive 
| Biotech and Pharmaceutical Executive 


“Quantifying Quality of Life: Incorporating Daily Life into Medicine is an innovative 
and informative text that explores technology-enabled assessments of specific 
dimensions of the WHOQOL framework. An underlying construct running through 
the text is the significance of quality of life, and its core role within clinical, health, 
and societal decision-making. Within this construct, the text brings together critical 
thinking across key concepts and domains of the International Classification of 
Functioning, Disability, and Health (ICF) to consider the emergent role of technology 
in measurement. The authors guide the reader through a series of chapters exploring 
measurement of quality of life domains through technology and innovation both 
with active self-report PROs and passively collecting physiological, physical, 
biological, and contextual signals. The text is rich with deep thinking by leading 
critical thinkers across domains of the ICF. Original work is freshly presented in the 
book's chapters, which draw on case-control to umbrella review methodologies. 
The authors all succeed in compellingly presenting the background, state of the art, 
and suggesting the direction of future research activity. The book is abundant with 
fine details and expertise across WHO domains to inform the quality of life research. 
The book provides a unique insight into an easily accessible template for 
multidisciplinary students, clinicians, and academics.” 
—Prof. Helen Dawes, Professor, 
Movement Science and Elizabeth Casson Trust Chair, Director, 
Centre for Movement and Occupational Rehabilitation Sciences (MOReS), 
Academic Director, Oxford Clinical Allied Technology 
and Trial Services Unit (OxCATTS), Faculty of Health and Life Sciences, 
Oxford Brookes University, Associate Research Fellow, 
Department of Clinical Neurology, 
University of Oxford, 
Oxford Health Biomedical Research Centre, UK 


“La transition vers l'hópital numérique est en cours à une époque oü l'on soigne de 
plus en plus à domicile, ou le patient prend le contróle de sa santé. La médecine 
jusqu'ici essentiellement curative, notamment en Occident, se transforme elle aussi 
avec une mise en exergue du patient: une personnalisée, préventive, prédictive et 
participative (4P). 

Nous sommes entrés dans cette nouvelle ére supportées par des technologies 
innovantes, comme le bigdata (analyse de données), le deep learning, la robotique, 
l'intelligence artificielle, le suivi de données personnelles, voir la génétique. 

Nous pourrons ainsi anticiper, prévenir, modifier notre hygiéne, non pas pour 
répondre à un modéle, mais à notre modéle personnalisé. Derriére cette 
personnalisation est en train de naître la médecine de trajectoires avec une prise en 
compte accrue de tout notre environnement (alimentation, qualité de l'air, pratique 
physique...). 

Le patient devient donc désormais l'acteur principal et participe à sa santé de 
manière active au travers de ses soins mais surtout d'une meilleure connaissance de 
sa santé à l'aide principalement d'applications (comme des protocoles de suivi post- 
cancer), ou de capteurs de données réguliéres ou alarmistes (IOT) et d'un accés 
facilité à l'information médical. Cette vision est à la base de ce livre, et il fournit de 
nombreux exemples d'innovations technologiques potentielles menant à la 
réalisation de cette vision. Ce livre est le fruit d'un effort remarquable pour 
documenter l'état de l'art sur les 24 variables contribuant à la qualité de vie dans ces 
chapitres bien présentés, clairs et complémentaires. Il est écrit de maniére à étre 
accessible à tous, méme aux patients. Je suis heureux de recommander cet excellent 
livre à tous!” 

—Moise Gerson, A Patient, Digital Health Director and Co-Founder, 
Campus 2030, INNOVATING FOR THE PEOPLE, Switzerland 


"Accurately assessing the quality of life (QoL) using established parameters, such 
as those set out by the WHO, is key to a revolution in the health and life insurance 
sectors. That is the use of real-time data to monitor risks and more accurately 
underwrite it using feedback loops. “Connected Insurance" is already in use in areas 
like the use of telematics for auto insurance and is similarly changing health and life 
insurance, helping insurers assist in preventing and managing chronic disease and 
acute incidents as well as encouraging healthier lifestyles in their customers. It has 
two important drivers. The first is that technologies for measurement, including 
mobile phone-based apps and wearables, are proliferating and improving and can 
allow real-time monitoring of many of the facets in the WHOQoL domains. The 
second is that accurate quantification of risk factors to scientific levels of accuracy 
(as demonstrated in these chapters) will provide increased statistical accuracy and 
confidence for underwriters of health and life risk. 
Quantifying the Quality of Life is an important mosaic of exciting research that 
shows how facets of QoL can be quantified." 
—Lawrence Reed, Managing Director, IMCG Group, UK 
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“Quantifying Quality Of Life: Incorporating Daily Life into Medicine is a must-read 
primer for anyone considering using patient-related outcomes with subjective and 
objective data. In health insurance, we are aware that healthy lifestyle choices like 
eating well, exercising, managing stress, and sufficient sleep can help mitigate the 
risks associated with lifestyle diseases. Therefore, we need to show clients how 
these different health factors are linked together. The power of ecosystems to 
develop a dynamic, personalized pricing approach that rewards good behaviors will 
potentially help to drive this forward. To achieve the next steps, it is essential to have 
valid, accurate, and reliable data for the QoL assessment for individuals. The book 
is full of important information for those of us who are on the journey to real patient- 
centricity by using relevant data outcomes.” 

—Roman Sauter, Head, Sourcing & Procurement, Helsana, Switzerland 


This series is directed to healthcare professionals leading the transformation of 
healthcare by using information and knowledge. For over 20 years, Health 
Informatics has offered a broad range of titles: some address specific professions 
such as nursing, medicine, and health administration; others cover special areas of 
practice such as trauma and radiology; still other books in the series focus on 
interdisciplinary issues, such as the computer based patient record, electronic health 
records, and networked healthcare systems. Editors and authors, eminent experts in 
their fields, offer their accounts of innovations in health informatics. Increasingly, 
these accounts go beyond hardware and software to address the role of information 
in influencing the transformation of healthcare delivery systems around the world. 
The series also increasingly focuses on the users of the information and systems: the 
organizational, behavioral, and societal changes that accompany the diffusion of 
information technology in health services environments. 

Developments in healthcare delivery are constant; in recent years, bioinformatics 
has emerged as a new field in health informatics to support emerging and ongoing 
developments in molecular biology. At the same time, further evolution of the field 
of health informatics is reflected in the introduction of concepts at the macro or 
health systems delivery level with major national initiatives related to electronic 
health records (EHR), data standards, and public health informatics. 

These changes will continue to shape health services in the twenty-first century. 
By making full and creative use of the technology to tame data and to transform 
information, Health Informatics will foster the development and use of new 
knowledge in healthcare. 


More information about this series at https://link.springer.com/bookseries/1 114 
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Part I 
Introduction 


Chapter 1 A) 
Unfolding the Quantification gese 
of Quality of Life 


Sharon Wulfovich, Jeppe Buur, and Katarzyna Wac 


Introduction 


There are many ways to define health. Health is defined by the World Health 
Organization (WHO) as “a state of complete physical, mental and social well-being 
and not merely the absence of disease or infirmity.” [1, 2] This definition has recently 
been challenged by a team of international experts who suggested that health be 
defined as “the ability to adapt and self-manage in the face of social, physical, and 
emotional challenges.” [3] Health contributes greatly to quality of life (QoL), and 
some authors suggest that health-related QoL and QoL can be used interchangeably 
[4]. However, QoL is more than health, as other factors including work capacity, social 
support, and the physical environment are also necessary for QoL [5-7]. QoL can be 
defined in multiple ways through a more global approach (from the psychological, 
economics, policy, or medical science perspective) [8], a categorical breakdown from 
an individual perspective (e.g., physical or psychological aspects), or a field-specific 
definition applied to individuals or specific populations (e.g., Liver QoL) [7, 9]. 
Across these different definitions, there is some agreement that QoL integrates 
an individual’s multidimensional evaluation of their own life and total well-being 
[7]. Furthermore, an individual’s QoL is not merely focused on the individual; it 
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encompasses the individual’s physical and psychological state, the environment the 
individual is in, as well as the interaction between the two. The environment includes 
other individuals; nonmaterial things such as parks and roads; as well as water, air, 
and access to other resources. 

Measuring an individual’s QoL allows us to obtain a more holistic assessment of 
his or her state in the multiple contexts like disease progression (via symptoms), or 
treatment progress, and to put that in the context of clinical decision making. QoL 
or well-being has been indirectly assessed since the dawn of the field of medicine. 
Almost every doctor or physician informally asks the patient about his or her state 
using questions such as “how are you feeling right now?" or “how are your 
symptoms?” 

With the need to systematically assess QoL in clinical decision making [10], 
there are two primary ways to capture this information: (1) asking people about dif- 
ferent aspects of their lives following subjective self-reporting using validated 
patient-reported outcomes (PRO) [11] instruments [12]; examples and an overview 
of the current validated instruments for QoL assessment can be found in the studies 
of Gill [13] and Linton et al. [14]; and (2) leveraging technologies to objectively 
capture individuals’ biological samples, physiological signals, behaviors, or inter- 
actions with the environment [4, 11]. 

One of the most widely used QoL assessment instruments is the WHO’s Quality 
of Life instrument (WHOQOL), which is used as a framework for organizing this 
book. The WHOQOL defines QoL as “individuals’ perception of their position in 
life in the context of the culture and value systems in which they live and in relation 
to their goals, expectations, standards and concerns.” [15] The WHOQOL-BREF 
instrument assesses individuals’ QoL across four domains: physical health, psycho- 
logical health, social relationships, and environmental aspects [16]. These four large 
domains are further broken down into 24 subdomains, denoted by the WHO as ‘fac- 
ets’ [15] (Fig. 1.1). The subdomains embrace subjective and objective aspects of life, 
are mutually nonexclusive, and potentially intertwine [15]. For example, there is an 
influence of noise (i.e., environmental aspect) on sleep and rest (1.e., physical health). 

The overarching assumption carried throughout this book is that within each of 
the QoL domains, there are specific daily behaviors that (a) can be accessed objec- 
tively through personal technologies or (b) enabled through the use of these tech- 
nologies. A behavior is defined by the scientific community as "internally 
coordinated responses (actions or inactions) of whole living organisms (individuals 
or groups) to internal and/or external stimuli, excluding responses more easily 
understood as developmental changes," [17] or as *a comportment, or what some- 
one does or how someone acts." [18] Behaviors can be assessed by means of, for 
example, their frequency, rate, duration, magnitude, and latency [19]. In the scope 
of this book, we focus specifically on external observable behaviors (or the lack 
thereof) that may be assessed using technologies. This assumption follows the defi- 
nition of QoL Technologies (QoLT) as “any technologies for assessment or improve- 
ment of the individual's QoL." [20] The variety of designs of QoLT used to assess 
behaviors in daily life remains unknown, as does their influence on QoL. In this 
book we focus solely on the approaches using technology-enabled QoL 
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QoL Domains QoL Subdomains 


Physical Health Activities of daily living 

Dependence on medicinal substances and medical aids 
Energy and fatigue 

Mobility 

Pain and discomfort 

Sleep and rest 

Work capacity 

Psychological Bodily image and appearance 

Negative feelings 

Positive feelings 

Self-esteem 

Spirituality/religion/personal beliefs 

Thinking, learning, memory and concentration 

Social Relationships| Personal relationships 

Social support 

Sexual activity 

Environment Financial resources 

Freedom, physical safety and security 

Health and social care: accessibility and quality 

Home environment 

Opportunities for acquiring new information and skills 
Participation in and opportunities for recreation/leisure act. 
Physical environment (pollution / noise / traffic / climate) 
Transport 


Fig. 1.1 WHOQOL Instrument Domains and Subdomains [16] 


assessments. We therefore assume that QoLT enable behavior assessments and as a 
result the assessment of QoL in individuals [11]. 

This book presents QoLT leveraged for QoL assessment and draws from the 
WHOQOL, providing a way to categorize behaviors and QoL aspects. As a result, 
the WHOQOL instrument presented here serves as the organizational method for 
this book. 

The remainder of this chapter is organized as follows: First, we present the 
WHOQOL instrument in detail (Sect. 2), and then we present the 'quantified- 
selfers’, who leverage daily life technologies to assess their own behaviors and daily 
life (Sect. 3). Lastly, we conclude with a discussion further motivating the vision for 
this book (Sect. 4). 

This book follows the WHOQOL instrument, and its chapters are organized 
along the WHOQOL subdomains. The following chapter discusses conclusive 
remarks and future directions for the field of QoL assessment. Finally, the last chap- 
ter discusses the current state of the subjective assessment of QoL by summarizing 
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a set of validated instruments and scales for assessing daily life behaviors in the 
context of QoL, also organized along the variables in the WHOQOL instrument. 


The WHOQOL Instrument 


The WHO developed its first edition of an international QoL assessment approach 
in 1995 [15]. The development of the WHOQOL consisted of many stages: (i) QoL 
concept clarification; (ii) qualitative pilot; (iii) development pilot; and (iv) field test 
[15]. Due to the multidimensional essence of QoL, the WHOQOL divided QoL into 
six broad domains: (1) physical domain; (2) psychological domain; (3) level of 
independence; (4) social relationships; (5) environment; and (6) spirituality/reli- 
gion/personal beliefs [15], embraced within the original 100-question instrument 
referred to as the WHOQOL-100 [21]. Later, the WHO developed a WHOQOL- 
BREF QoL assessment [16, 22], a shorter version of the original WHOQOL-100, 
which defines four broad domains: (1) physical health; (2) psychological health; (3) 
social relationships; and (4) environment [16, 22]. This shorter version, used as the 
assessment model throughout this book, was developed to minimize respondent 
burden and unnecessary detail when approaching QoL assessment in the general 
population [16, 22]. The WHOQOL-BREF instrument has been demonstrated to 
have “good to excellent psychometric properties of reliability” and to perform “well 
in preliminary tests of validity.” [22] 

The WHOQOL-BREF, its four domains, and the 24 subdomains are outlined in 
Fig. 1.1. The paragraphs below provide a working definition for each of the four 
domains and 24 subdomains of the WHOQOL-BREF. We use the titles of the 
WHOQOL-BREF and have adapted the definitions of the WHOQOL User manual 
[23] accordingly, as the WHOQOL-BREF does not have its own manual. 


Domain I: Physical Health 

1. Activities of Daily Living: “a person’s ability to perform usual daily living 
activities.” 

2. Dependence on Medicinal Substances and Medical Aids: “a person's depen- 
dence on medication or alternative medicines for supporting his/her physical and 
psychological well-being." 

3. Energy and Fatigue: “the energy, enthusiasm and endurance that a person has in 
order to perform the necessary tasks of daily living, as well as other chosen 
activities such as recreation." 

4. Mobility: “the person's view of his/her ability to get from one place to another, 
to move around the home, move around the workplace, or to and from transpor- 
tation services." 

5. Pain and Discomfort: “unpleasant physical sensation experienced by a person 
and the extent to which these sensations are distressing and interfere with life." 
The topics include pain control. 

6. Sleep and Rest: problems getting enough sleep and rest. 
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7. Work Capacity: “a person's use of his or her energy for work.” “Work” is defined 
as any major activity in which the person is engaged. 


Domain II: Psychological Health 
8. Bodily Image and Appearance: “the person's view of his/her body.” 
9. Negative Feelings: “how much a person experiences negative feelings, includ- 
ing despondency, guilt, sadness, tearfulness, despair, nervousness, anxiety and 
a lack of pleasure in life." 

10. Positive Feelings: “how much a person experiences positive feelings of content- 
ment, balance, peace, happiness, hopefulness, joy and enjoyment of the good 
things in life." 

11. Self-Esteem: “how people feel about themselves." 

12. Spirituality/Religion/Personal Beliefs: “examines the person's personal beliefs 
and how these affect quality of life." 

13. Thinking, Learning, Memory, and Concentration: “a person's view of his/her 
thinking, learning, memory, concentration and ability to make decisions." 


Domain III: Social Relationships 

14. Personal Relationships: “the extent to which people feel the companionship, 
love and support they desire from the intimate relationship(s) in their life." 

15. Social Support: “how much a person feels the commitment, approval, and avail- 
ability of practical assistance from family and friends." 

16. Sexual Activity: “a person's urge and desire for sex, and the extent to which the 
person is able to express and enjoy his/her sexual desire appropriately." 


Domain IV: Environment 

17. Financial Resources: “the person's view of how his/her financial resources (and 
other exchangeable resources) and the extent to which these resources meet the 
needs for a healthy and comfortable life style." 

18. Freedom, Physical Safety, and Security: "the person's sense of safety and secu- 
rity from physical harm.” 

19. Health and Social Care: Availability and Quality: “the person’s view of the 
health and social care in the near vicinity.” 

20. Home Environment: the “principal place where a person lives, and the way that 
this impacts on the person’s life. The quality of the home would be assessed on 
the basis of being comfortable, as well as affording the person a safe place to 
reside.” 

21. Opportunities for Acquiring New Information and Skills: “a person’s opportu- 
nity and desire to learn new skills, acquire new knowledge, and feel in touch 
with what is going on.” 

22. Participation in and Opportunities for Recreation/Leisure Activities: "a per- 
son's ability, opportunities and inclination to participate in leisure, pastimes and 
relaxation." 

23. Physical Environment (Pollution/Noise/Traffic/Climate): “the person's view of 
his/her environment. This includes the noise, pollution, climate and general 
aesthetic of the environment and whether this serves to improve or adversely 
affect quality of life." 
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24. Transport: “the person’s view of how available or easy it is to find and use trans- 
port services to get around.” [23] 


The WHOQOL instrument provides a way to categorize behaviors and QoL 
aspects. As defined earlier, the WHOQOL instrument presented here serves as the 
organizational method for this book. 


Learning from the ‘Quantified-Self’ Community 


In this section, we present and discuss a subset of currently available technologies 
for the assessment of behaviors, health state, and—as a result —QoL. The “quanti- 
fied self" is a term coined in 2007 by Gary Wolf and Kevin Kelly to accommodate 
actions such as lifelogging and self-tracking, in which motivated individuals use 
various analogues (e.g., paper and pencil) or digital, technology-enabled tools (e.g., 
Excel spreadsheets) and devices (e.g., wearables) for tracking certain aspects of 
their lives—be they in relation to physical health, mental health, social relation- 
ships, or even the environment surrounding them. This section presents a nonex- 
haustive view on the community, surveying individuals who actively participate in 
the quantified-self movement, as well as what they self-track and how they do it. 
Specifically, this chapter presents a qualitative study that examined the quantified- 
self community based on a curated set of self-tracked projects presented in video 
talks from quantified-self conferences and meetings (organized in the form of meet- 
ups) between 2015 and 2019. In total, 71 quantified-self projects were analyzed 
with the purpose of finding out who self-tracks, what they track, and how they track 
it. A variety of variables are categorized and analyzed, including the self-tracker's 
sex, domains of tracking (coded along the WHOQOL instrument dimensions and 
subdomains), and devices and tools used, among others. We then extrapolate upon 
the applicability of the tools, approaches, and lessons learned toward the larger 
public, for which we aim to quantify QoL. 

This section is structured as follows: we first define the quantified-self movement 
and quantified-self community, then describe our research methods and results, and 
then analyze the outcomes, implications, and limitations. We end with a conclusive 
remarks section that summarizes the lessons learned within the section and links 
them to the chapter as a whole. 


The Quantified-Self Movement 


The quantified-self is a way of logging and measuring a variety of data about an 
individual and/or his/her surroundings, such as steps, calories eaten, or miles biked 
[24]. The quantified-self in its simplest form is a way of logging a variety of data for 
different reasons, be it for self-improvement, curiosity, or health benefits related to 
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a specific tracking category. It is a practice that has developed a particularly rich 
design space with the introduction of personal digital technologies enabling self- 
tracking, such as smartphones, smartwatches, and intelligent wristbands that, among 
other devices, are now a part of many individuals’ lives. Yet, the actual practice, 
emergence, and use of self-tracking as a method have been discussed for millennia 
[24]. While most individuals may or may not be aware of technology such as step 
counters in smartphones, the population of quantified-selfers purposely tracks an 
array of different variables of their lives, both quantitatively and qualitatively, with 
various goals, thus contributing to an enhanced understanding of their own behav- 
iors, state, and potentially their own QoL. 


The Quantified-Self Community 


The quantified-self community arranges conferences and meet-ups where quantified- 
selfers have the opportunity to present their individual projects. Their motto is "se/f- 
knowledge through numbers.” [25] This community provides a platform for 
individuals—in principle anyone—to present what they did, how they did it, and 
what they learned, from which other individuals can both learn and be inspired to 
shape their own projects [26]. Throughout the year, location-based group meet-ups 
are conducted within the quantified-self community. Furthermore, yearly (or bi- 
yearly) conferences are held in which individuals from all over the world participate 
[25]. The talks at the conferences are recorded, and the best of which (as selected by 
the community founders) are curated and published on the community website. 
These video recordings are the primary self-project materials leveraged for the anal- 
ysis in this section. 


Methods 


This section provides insights into the methodological considerations of this study 
as well as a justification of methodological choices, as there will arguably be differ- 
ent ways to interpret and work with the self-tracking project material gathered by 
the quantified-self community in both current and future research. 

The research in this section largely follows a qualitative research methodology 
that incorporates the basic principles of hermeneutics [27], which ensures that both 
the data and their interpretation are conducted cautiously. For this section, it is cru- 
cial to examine these results with a pre-understanding of self-tracking projects 
being both inductive and deductive in nature, and to acknowledge a potential confir- 
mation bias in the self-tracking projects. As with most qualitative research, given 
this approach, it is difficult to generalize the results for the specified population, but 
the goal is not to develop a standing thesis about the quantified-self community; 
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rather, the goal is to identify current self-tracking trends and patterns within the 
sample [28]. 

The self-tracking projects analyzed in this section are in the form of video record- 
ings of a talk related to the project, as presented by its author at a quantified-self 
conference. The employed video sampling method was purposive sampling. This 
represents several approaches within purposive sampling in qualitative research 
methods [28]. The video inclusion criteria included the following: the video mate- 
rial had to revolve around a quantified-self project and had to be selected by the 
community leaders to be uploaded to the quantified-self website, thus narrowing the 
analyzed examples to those presented within the curated content. Therefore, some 
of the examined cases are arguably extreme deviant cases, rather than typical self- 
tracking project cases. 

This research was approached as a bottom-up gathering of data, from which 72 
video presentations from quantified-self conferences (2015-2019) were examined 
and analyzed based on pre-existing themes and categorizations, as well as on themes 
growing from the material while trawling video presentations. Without adding 
search filters, the website was trawled from the top (the newest) to bottom, within 
the timeframe for the study (14 weeks in total). The website was updated twice 
(week 10 and week 14 of 2019) during the project, which means the order of the 
videos examined was disrupted at least these twice, and new video presentations 
were added during the study period. One video was deemed to be outside the scope 
of the project due to it having a vastly different goal to the others (i.e., educational), 
in which a use case of “quantified-self’ as an educational material was presented. 
This video was removed from the data set, thus making the actual data set consist of 
71 videos. An overview of the 71 analyzed self-tracking projects (authors, titles, and 
years of publication) can be found in Appendix 1. 

While it may be difficult to fully transform the words and personal experiences 
of self-trackers into quantitative evidence, we aimed to present here the qualitative 
approach that we employed; categorization and thematic analysis provided the 
opportunity to count self-tracking projects and partly quantify some of the material 
presented within each project [28]. The thematic analysis is based on examining 
project descriptions and identifying two kinds of codes to describe the projects: 
descriptive and interpretive codes [29]. The interpretive codes were defined before- 
hand (along the WHOQOL subdomains), and the descriptive codes were identified 
afterwards and noted within the dataset, as presented later in this section. The codes 
were agreed upon by two independent coders. 


Results 


First, the overall findings are presented per a WHOQOL variable (Sect. 3.4.1), and 
afterwards a deeper examination is presented of what is tracked and how it is done, 
starting at a macro level and then proceeding into a micro level analysis of different 
variables included in each of the projects (Sect. 3.4.2). 
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One-Dimensional Presentation of Data and Findings 
Sex Self-tracker sample comprised n = 26 females (36%) and n = 45 males (64%). 


QoL Domains Figure 1.2 presents the coded self-tracking project along the 
WHOQOL-BREF domains. Each project was assigned one main QoL domain, even 
if, as presented below, some projects in fact analyzed two or more different subdo- 
mains. As seen in Fig. 1.2, the largest domain that was tracked is the physical 
domain, embracing variables such as “exercise” and “sleep,” as detailed later. The 
distribution of the tracked QoL domains is as follows: physical (n = 41), psycho- 
logical (n = 26), social (n = 1), and environmental (n = 3). 


QoL Subdomains Figure 1.3 presents the domain distribution with the subdo- 
mains. It is important to keep in mind that several projects are marked with more 
than one code, due to the projects sometime being cross-field examinations of parts 
of the self-tracker’s life, or even holding variables from different domains against 
each-other, such as “location tracking" as a facilitation of “memory tracking of 
daily life activities," which in WHOQOL codes corresponds to *env-environ" (loca- 
tion) being tracked to keep track of “phy-adl” (activities) (appendix 2) [30]. A total 
of 84 codes were applied across 71 videos and are presented in Fig. 1.3 below. 


Quantitative Vs. Qualitative Project Each project was assigned to one category 
(quantitative or qualitative) depending on its main goal. While the community is 
named after the term ‘quantified self’, qualitative studies are also present in the 
sample. Studies that have been defined as quantitative rely on datasets derived from 
smartphone apps (e.g., location) or wearables (e.g., steps). Studies that were defined 
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Fig. 1.3 Distribution of the WHOQOL Subdomains Tracked 


as qualitative relied on journals/diaries or other kinds of self-reporting tools to 
describe feelings/emotions or other internal, difficult-to-observe states of individu- 
als. Overall, within the sample projects, quantitative projects (n = 59) were more 
popular than qualitative projects (n = 12). 


Manual Vs. Automatic Tracking Each self-tracking project has an element of 
tracking quantitative and/or qualitative data, and this tracking can be realized 
through manual (e.g., paper and pencil) or automatic means (e.g., smartphone phone 
loggers). Each project was assigned to one category (manual or automatic) depend- 
ing on its main goal. The results reveal that the majority of projects include auto- 
matic logging (n = 42), whereas the projects with manual logging (via e.g., a 
spreadsheet) are less represented (n = 29). 
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Observations 


Fig. 1.4 Projects’ Self-Tracking Frequency 


Tracking Frequency Each self-tracking project has an element of tracking of 
some data at a specific frequency (from ‘one-off’, i.e., one observation, not repeated, 
to repeated ‘daily’ to less frequent), and it was assigned to a category depending on 
its main goal. Figure 1.4 presents the tracking frequency, whether it is daily (n = 53), 
weekly (n = 5), monthly (n = 5), yearly (n = 1) or one-off (n = 7). The daily group 
is the largest group, followed by one-off projects, weekly, monthly, and finally 
*yearly' tracking. 


Self-Tracking Project Duration It was difficult to analyze project duration 
because many projects contained no clear indication of their length and were thus 
coded as ‘N/A’ (n = 54). This is due to a variety of reasons, but most commonly it 
seemed that some projects did not focus on events in a given time duration, but 
rather on a number of certain events to be tracked in some (unspecified) observa- 
tional period, selected as convenient, or even defined only post-experimentally by 
the individual. However, it can be noted that the most common durations range from 
1 month (n = 3), 1 year (n = 7), and 3 years (n = 4) to 10 years (n = 3). 


Observational Vs. Interventional Projects With regard to whether the project 
was an observation of an individual state or behavior, or explicit intervention (imply- 
ing an implicit intention of change of an individual state or behavior), it was found 
that observations were the most common aim of individual self-tracking (n = 47). 
Interventions were documented within 22 projects. The last two projects led from 
Observation to intervention on the state or behavior observed at first. 


Self-Tracking Tools Used A total of 71 unique commercial and noncommercial 
devices and digital tools were identified through the course of this study. For the 
sake of simplicity, both actual devices such as wearables and smartphones (and their 
apps) were defined as a “self-tracking tool." Additionally, these seem to have 
increasing importance in the quantified-self community as well as in everyday life 
[24, 31]. Furthermore, several projects relied on data provided from companies such 
as 23andme and uBiome, which were also defined as a tool for self-tracking in this 
project. Several projects used multiple tools to gather their desired data. Figure 1.5 
illustrates the distribution of the various tools across all 71 projects in a diagram, 
where the tools written in larger fonts correspond to the more common tools and 
those in smaller ones correspond to less common tools. The color coding is arbitrary. 
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Fig. 1.5 Tools Used in Self-Tracking Projects 


The most commonly used tool was a spreadsheet (n = 7), which has many affor- 
dances with regards to data. It allows for data manipulation and statistical analysis 
and cooperates well, for example, with self-written analytics scripts (e.g., Python) 
and programs. Other popular tools were wearable devices such as Fitbit (n — 5). 
These provide basic biometric information, such as current heart rate and sleep 
schedule as well as an activity counter [32]. The Freestyle Libre (n = 5) was another 
popular tool in this sample, which is a continuous blood glucose monitor (CGM) 
essentially developed for diabetics to minimally invasively monitor their blood glu- 
cose levels. Due to its ease of use, availability for ‘over-the-counter’ sale, and 
affordable price, nondiabetics also use it [33]. 


Two-Dimensional Presentation of Data and Findings 


Sex Vs. Self- Tracked WHOQOL Domain As seen in Fig. 1.6, the sex distribution 
analyzed against the WHOQOL domains illustrates that an imbalanced distribution 
exists for physical health tracking (with more male trackers) and an even level of the 
tracking of psychological aspects of life, even though male presenters represent the 
vast majority of the sample overall (n 2 44). It is important to consider these results 
for the population analyzed within this section, rather than as results that can be 
generalized over a wider population of self-trackers. 


Sex Vs. Self- Tracked Variable The top portion of Fig. 1.7 presents the distribution 
of all the self-tracking variables among the projects. The variables written in larger 
fonts correspond to the most common ones, and those in smaller fonts correspond 
to less common variables. The bottom portion of the figure presents the distribution 
of self-tracking variables (i.e., independent variables) sorted by the tracker's sex. 
There were n = 46 independent variables identified as categories describing the 
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self-tracking project and categorized under the four main WHOQOL domains. 
There were relatively few repeated variables, but “productivity” (n = 5) was the 
most observed variable for female trackers, whereas "sleep" (n 2 4) was the most 
observed variable for male trackers. Furthermore, *menstrual cycle" (n 2 3) was the 
second most tracked independent variable for female trackers, whereas “daily activ- 
ities” (n = 3), “running” (n = 3), and "stress" (n = 3) shared this position for male 
trackers. The rest of the variables were unique for one or two projects (n = 40 
projects). 


Discussion 


While the results are not generalizable, they do prove one point: the field and inter- 
est of the quantified-self projects and inputs to the community are highly diverse 
and represent a broad spectrum of self-tracking projects. 

The distribution of self-trackers’ sex is interesting to reflect upon. It suggests that 
female self-trackers would perhaps be more inclined to conduct a self-tracking 
study with an emphasis on psychological means of life, whereas male trackers seem 
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Fig. 1.7 Self-Tracked Independent Variables (top) Mapped by the Sex of Self-Trackers (bottom) 


more inclined toward physical and environmental studies. While these results do not 
necessarily say anything absolute about the correlation of these variables, it is an 
interesting perspective on the distribution of sex vs. domain-related projects as well 
as personal self-interest. 

As for the tools utilized within the self-tracking projects, they varied as greatly 
as the variables that were tracked. There was a relatively high number of observa- 
tions for tools such as simple spreadsheets. The most tracked WHOQOL category 
was the physical health domain, while at the subdomain level it was “activities of 
daily life"—Aa broad category that accommodates numerous types of activities and 
diverse tracking approaches depending on the type and frequency of activity tracked. 
The self-tracker community is also interested in tracking variables that are not yet 
available for autonomous, pervasive tracking leveraging digital tools (e.g., moods 
and mental states). This can be seen when analyzing the lesser categories—the 
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psychological health domain and more specifically the “thinking, learning, memory, 
and concentration” variable, which has high interest and includes self-tracking of, 
for example, mindfulness and willpower. Again, to date, these are almost impossible 
to track autonomously and pervasively in the daily life of individuals. Our results 
indicate that the projects are not necessarily dictated by which self-tracking tools 
exist, but rather by curiosity and personal interest in self-tracking. 

The results acquired here also provide an interesting perspective on the quantified- 
self as a trend itself, since on the one hand we are living in an age where we are 
"datafying" ourselves at an increasing rate, while on the other hand data protection 
and privacy questions are arising with the digitalization of everyday means [24]. 
These questions will become even more urgent to tackle with the emergence of tools 
tracking psychological health [34, 35]. 

In addition, the acquired results bring into question the ultimate goal of self- 
tracking. The idea of the quantified-self stems from the idea of converting aspects of 
life into numbers and statistics, rather than (qualitative) writing in a daily journal to 
keep up with life. This can be described as the aspect of self-betterment, in which 
individuals seek to better understand themselves through numbers and analyses of 
everyday actions [24]. With roughly two out of three projects being observational in 
nature, this does not mean that the individuals involved do not seek behavior change, 
but it may be a distant goal rather than an immediate need. The immediate need 
focuses on understanding factors influencing one’s own behavior and state. Few 
cases have provided evidence that the results of an observation could be transformed 
into an intervention, specifically when the results have been too crucial to ignore for 
the individual self-tracker. One such case was a male individual who felt upset with 
drivers looking at their personal devices while driving. He decided to investigate his 
own behavior while driving and set up a quantified-self project to help him reach his 
goal of spending even less time looking at his phone while driving. He hypothesized 
that the results would indicate that he was much better than other drivers; however, 
that was not the case, as he realized he spent up to 25% of his time while driving on 
his phone (up to around 25 minutes per a day along a 100-minute commute). This 
observation called for an immediate intervention, and this self-tracker ultimately 
bought a bike for smaller trips—which also implied he was getting some physical 
activity while commuting (and not using his phone) [36]. This is just one example 
of a self-tracking project—including meticulous observation and self-reflection— 
turning into an intervention. 


Limitations 


An array of limitations arose in this study, which stemmed from the methodological 
approach as well as the approach to data analysis. This section presents some of 
these limitations. 

At first, the coding and categories examined in the study were predefined, par- 
tially based on the WHOQOL and only loosely based on existing literature. The 
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other categorizations (the study duration and tools used) were agreed upon between 
two independent coders (having 90% agreement) as to what could prove to be inter- 
esting for the domain of QoLT, rather than what was found interesting in previous 
studies informing the QoL domain. This led to several categories yielding insuffi- 
cient results or not covering relevant aspects of the research. Two examples are 
additional coding dimensions discussed along the “work/leisure” category (i.e., the 
professional or personal domain aim of the project) and the “chronic illness” cate- 
gory (i.e., if the self-tracker was a patient). It was proposed that, when identifying 
relevant categories, projects related to either work or leisure could yield interesting 
results; instead, it was almost impossible to define whether a project was solely 
based on or related to individuals’ work or leisure activities, and most categoriza- 
tions ended up being a mixture of the two, which were then omitted. The self- 
trackers approached their self-tracking projects—as well as their own 
lives—holistically, and the projects encompassed these two domains. It was also 
proposed that chronic illness of the self-tracker, if applicable to his or her condition, 
could prove interesting to examine, especially if it was explicitly stated to be a 
major part (and potentially part of the aim) of the project. It turned out that just five 
out of 71 projects were based on chronic conditions, thus making it difficult to 
examine this dimension thoroughly. 

We only analyzed a small percentage of the whole set of self-tracked proj- 
ects (using the video database of quantified-self talks) that could have been 
examined. Our material only provides a narrow view of the overall population 
and its recent projects and cannot be generalized. Overall, the nature of qualita- 
tive studies makes it difficult to replicate their results, as the qualitative under- 
standing and perception of material may differ in the “eyes of the analyzer.” [28] 

This research does not derive or even suggest correlations between multiple vari- 
ables, which could have proved to be an interesting aim on its own and should be 
considered in future work in the field, especially when larger datasets are acquired. 
Another limitation with regards to multivariate analysis is a lack of acquisition of 
basic information about the individuals studied. The only personal information col- 
lected is sex, which does not distinguish level of education, age, country of origin, 
cultural background, attitudes, or motivations for specific self-tracked variable(s), 
nor does it distinguish the level of digital literacy, which may be of importance 
when discussing the tools employed and use of data. These variables would have 
been paramount to include in an actual multivariate analysis, but it has not been 
possible to include them due to the structure of the datasets (i.e., the data were 
derived from videos). 


Concluding Remarks on Self-Tracking and Quality of Life 


While it is still too early to conclude anything that could be generalized to the 
population of the quantified-self community, which could then be applied to 
quantifying QoL, several valuable observations should be noted. Male 
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individuals were still dominant in the sample of self-trackers, which was also 
presupposed [24]. The projects have mostly focused on the physical domain of 
QoL, whereas the social domain has been focused on the least. Tools enabling 
automatic tracking of variables have been more commonly used as a method for 
collecting data within the projects. At a more specific level, physical health— 
“activities of daily life” is the most tracked subdomain across all QoL domains, 
with the next most common being psychological health— "thinking, learning, 
memory, and concentration.” Overall, individuals track many different categories 
of their lives and only a few variables are more dominant than others, namely 
“sleep,” "stress," and “running” are the dominant variables for males, whereas 
“productivity” and “menstrual cycle" are for females. In total, 47 projects were 
observational, and thus had no inherent goal of changing the behavior of the indi- 
vidual, and 22 projects were designed to be behavioral interventions from the 
beginning. Two observational projects led to interventions. These behavioral 
interventions were self-designed and self-tracked and led to sustainable behavior 
change in most cases. 

What we are able to derive from self-trackers is that their attitude, motivation, 
and overall curiosity-driven and personalized approach are likely to lead to effective 
change and improvement in the understanding of factors influencing the behavior or 
state, or to sustainable change in this behavior. 


Future Outlook: Importance of Improving Quality 
of Life Quantification 


Quality of life is a critical outcome in daily life and in medicine. Long-term QoL 
stems from behaviors and states that are repeated frequently; therefore, long-term 
QoL may be extrapolated through the quantification of these (short-term) behav- 
iors/states. The quantified-self community's efforts illustrate that we can leverage 
various existing and emerging tools to observe and understand our own behaviors 
and states, and improve them through self-management as well as meticulously 
designed, highly personalized interventions. An integral part of future research 
on QoL technologies and their use in medicine is an interdisciplinary effort for 
achieving a user-centric and holistic approach, including physical, psychological, 
social, and environmental viewpoints. An interdisciplinary approach is required 
because assessment and management of behavior in medicine cannot be readily 
completed using solely one of the dimensions (e.g., physical) or by one system- 
atic methodological approach (e.g., qualitative or quantitative). Holistic individ- 
ual assessment and improvement research will bring new approaches to theory, 
design, methods, measurement, and data analysis specific to each dimension, thus 
deepening it while enabling breadth. Because of the technological and method- 
ological advances required, such research is a long-term process rather than a 
short-term self-contained activity. 
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The QoLT field is in its nascent stage. This book presents an overview of the 
state-of-the-art methods and tools for quantifying daily life, health, and QoL state 
through QoLT across all the QoL domains and subdomains. An enhanced under- 
standing of technology-enabled or -supported continuous assessment methods of 
behaviors and states will allow for an improved understanding and modeling of the 
short- as well as long-term health and QoL of individuals. 


Appendices 

Appendix 1 
Quantified-Self Talks (Author Name: Title, Year) Year 
Steven Jonas: Stressing out loud 2013 
Kendra Albert: The great book project of 2013 2014 
Valerie Lanard: Breaking the TV habit 2015 
Jim McCarter: Effects of a year in ketosis 2015 
Ilyse Magy: Know thy cycle, know thy self 2016 
Ellis Bartholomeus: Draw a face a day 2016 
Robert Macdonell: The data is in, I am a distracted driver 2016 
Ahnjili Zhuparris: Menstrual cycles, 50 cents and right swipes 2016 
Randy Sargent: Unlocking patterns with spectograms 2016 
Richard sprague: Microbiome gut cleanse 2016 


Peter Torelli: Narratives hidden in 20 years of personal financial | 2016 
data 


Abe gong: Changing sleep habits with unforgettable reminders 2016 
Mark Leavitt: Daily HRV as a measure of health and willpower 2016 


Akhsar Kharebov: A smart scale for healthy weight loss 2016 
Shelly Jang: Can you see that I was falling in love? 2016 
Steven cartwright: /7 years of location tracking 2016 


Paul Lafontaine: Using heart rate variability to analyze stress in | 2016 
conversation 


Jon cousins: Why I weighed my whiskers 2016 
Mark Wilson: Three years of logging my inbox 2016 
Bethany Soule: Extreme productivity 2016 
Jacek Smolicki: Self-tracking as an artistic practice 2016 
Robby Macdonell: The data is in, I am a distracted driver 2016 
Randy Sargent: Unlocking patterns with spectograms 2016 


Thomas Christiansen: Over-instrumented running: What I learned | 2017 
from doing too much 


Ahnjili ZhuParris: Finding my psychedelic sweet spot using R 2017 


Stephen cartwright: Seeing my data in 3d 2017 
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Quantified-Self Talks (Author Name: Title, Year) Year 
Whitney Erin Boesel: My numbers sucked but I made this baby 2017 
anyway 

Kyril Potapov: Tracking productivity for personal growth 2017 
Lillian Karabaic: What if my life was the economy of A small 2017 
country? 

Sara Riggare: Balancing neurotransmitters in neurological illness | 2017 
Ellis Bartholomeus: My health scars 2017 
Robin Weis: Crying 2017 
Azure Grant: Hot stuff: Body temperature and ovulatory cycles 2017 
Justin Lawler: Taking on my osteoporosis 2017 
Azure Grant: My biological rhythms in sickness and in health 2018 
Thomas Blomseth Christiansen: Which grasses aggravate my 2018 
allergies? 

Mikey Sklar: Three marathons on zero calories 2018 
Justin Lawler: Tracking glucose as A person without diabetes 2018 
Madison Lukaczyk: How work distractions affect my focus 2018 
Whitney Erin Boesel: Cholesterol levels while nursing 2018 
Benjamin best: My blood values from diet and other activities 2018 
Albara Alohali: Running storytelling 2018 
Lydia Lutsyshyna: Separating work and home 2018 
Benjamin Smarr: Does my stomach anticipate my meals? 2018 
Shamay Agaron: Tracking breathing to control my focus 2018 
Maggie Delano: Quantifying my Phd: Pomodoros and 2018 
productivity 

Jessica Ching: Learning an impossible form of exercise 2018 
Kyrill Potapov: What Insidetracker taught me about my five-day | 2018 
fast 

Daniel reeves: Tracking my personal reliability 2018 
Fah Sathirapongsasuti: Blood oxygen on Mt. Everest 2018 
Mad ball: A self-study of my Child's genetic risk 2018 
Aaron Parecki: Ten years of tracking my location 2018 
Aaron Yih: Tracking across generations 2018 
Alec Rogers: What I'm learning from my meditation app 2018 
Jordan Clark: Quantifying the effects of microaggressions 2018 
Jakob Eg Larsen: My headaches from tracking headaches 2018 
Todd Greco: Building my external brain 2018 
Lillian Karabaic: Z/00daysofqs: Daily art from data 2018 
Ralph Pethica: Finding the optimal training zone 2018 
Anna Franziska Michel: Using running and cycling data to 2018 
inform my fashion 

Eli Ricker: Tracking what I do versus what I say I'll do 2018 
Shara Rags: Estrogen and invention 2018 
Stephen Maher: A decade of tracking headaches 2018 
Valerie Lanard: Learning from excuses 2018 
Eric Jain: Four weeks of blood sugar tracking 2019 
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Quantified-Self Talks (Author Name: Title, Year) Year 
Kyrill Potapov: Finding my optimum Reading speed N/A 
Rocio Chongtay: Quantified brain and music for self-tuning N/A 
Mark Drangsholt: What causes my heart rhythm disorder N/A 
Steven Jonas: Memorizing my daybook N/A 
Steven Jonas: Spaced listening N/A 
Ari Meisel: Experiments in treating my Crohn’s disease N/A 
Appendix 2 


WHOQOL Codes used for categorizing projects from the quantified-self commu- 
nity, following past work of Wac [37]. 


QoL Domain 
‘Phy’: Physical health 


Subdomains 
Phy-adl, phy-meds, phy-energy, phy-mobility, phy-pain, phy-sleep, 
phy-work 


*Psy":Psychological Psy-bodyimage, psy-negativefeel, psy-positivefeel, psy-selfesteem, 

health psy-beliefs, psy-thinking 

*Soc':Social relations | Soc-relationships, soc-support, soc-sex 

‘Env’:Environment Env-finances, env-freedom, env-healthcare, env-home, env-info, 
env-leisure, env-environ, env-transport 
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Mirana Randriambelonoro 


Introduction 


Developed countries are facing the challenge of ageing societies, lack of infrastruc- 
ture for healthcare and high cost of care. Researchers have been attempting to 
answer these problems by using innovative technology to promote healthy ageing 
[1]. Rather than the absence of disease, “healthy ageing” is defined as a process that 
enables older people to continue to do the things that are important for them such as 
performing activity of daily living, maintaining social contact and conserving dig- 
nity [2-4]. 

Activities of Daily Living (ADLs) include individual’s basic physical needs such 
as dressing, feeding, personal hygiene, continence and transferring [5]. To assess 
individual's level of independence, researchers and clinicians often measure the 
ability to perform ADLs' tasks through different approaches such as self-report and 
performance-based measures [6]. However, these methods which are commonly 
completed by caregivers are not completely free of bias and are sometimes subject 
to under or overestimation of the individual's true functioning [7]. 

Besides, the Quantified Self (QS) movement is gaining more and more attention. 
Itis an emerging trend, which allows individuals to self-monitor or self-track their 
daily life activities, to analyze and self-reflect on their behaviors, and to bring poten- 
tial change in their daily habits [8, 9]. Furthermore, with the recent development of 
monitoring technologies and artificial intelligence techniques, the possibilities to 
automatically distinguish between different ADLs and detect unexpected events 
such as falls, have become an interesting, important and potentially impactful topic 
[10]. How efficient these tools are, to assess and improve ADLs, needs further 
investigations. 
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This chapter will present the standard validated scales for ADLs and the current 
researches on the use of technologies to assess one’s ability to perform ADLs, 
mainly indoor-outdoor mobility and nutrition. To expand our overview on the topic, 
the focus will be on elderly as well as on younger and healthier individuals. We then 
follow with a nutrition use case assessment based on a conversational agent called 
"MiranaBot". 

Conclusive remarks will emphasize the necessity to consider behavioral science 
along with cultural and environmental context to elaborate personalized monitoring 
and intervention strategies, which will be translated to innovative solutions to pro- 
mote independence at home. 


State of the Art 
Activities of Daily Living (ADLs) 


Created by Sidney Katz in 1950 [11, 12], the term "Activities of Daily Living" 
refers to the fundamental skills required to care for one-self and live independently. 
ADLs include dressing, feeding, personal hygiene, continence and transferring [5]. 
ADL dependence is often associated with lower quality of life, higher healthcare 
costs, higher risk of mortality and institutionalization [13—15]. Assessing individu- 
al's ADLs is therefore essential, especially for vulnerable population who may need 
assistance in performing these activities. In this chapter, the focus will be on indoor/ 
outdoor mobility and nutrition. 

Mobility is defined as the ability to move oneself from one point (or state) to 
another inside and outside the home environment to maintain a certain level of inde- 
pendence [16, 17]. As activity restriction is associated with various physical decon- 
ditioning effects and decreased rates of social involvement, mobility is closely 
linked to health status and quality of life. Researchers has been attempting to pro- 
pose a mobility model based on multiple factors such as cognitive, social, physical, 
environmental, financial as well as gender, culture and age, which all may influence 
individual's mobility in different way [16]. As people age, decreased mobility is 
often predictor of falls and physical disabilities [18]. Consequently, there is a grow- 
ing interest in measuring mobility and determining the factors that influence mobil- 
ity to strengthen the physical and functional capacities of older adults. 

Nutrition capacity or the ability to eat is a part of the essential ADLs to maintain 
an independent life [5]. Food intake plays a crucial role in individual's life and 
momentary, as well as long-term health and quality of life. Research on nutrition sci- 
ence, human nutrition, preventive nutrition, clinical nutrition and public health nutri- 
tion has been specifically on the rise the past years [19], to treat and prevent medical 
conditions such as diabetes [20] or sarcopenia [21] and ensure a better quality of life 
[22]. Reliably measuring not only the quantity but also the quality and the frequency 
of food intake is therefore very important to picture one's nutritional habits and to be 
able to recommend adapted and personalized advice or treatment to the person. 
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Validated Scales for ADLs 


Several methods, such as self-report, caregiver report, or direct observation, have 
been used to measure the degree of independence in ADLs. Self-reporting is easy to 
perform and are often used when direct observation is not possible, or if the person 
presents no cognitive deficiency. Nevertheless, the self-reports may be biased and 
less valid when ones have difficulties to evaluate their functional capacity [23, 24]. 
Performance-based assessments imply in-lab assessment of the individual’s ability 
and skills for self-care using mock settings, role play and simulation of real-world 
environment. They may provide accurate information about the ability to execute 
the ADLs but typically require more qualified assessor to be performed, compared 
to self-reporting or informant reports. Table 2.1 presents four main validated scales 
widely used to measure ADLs in older adults or patients with cognitive or mobility 
impairment as well as their benefits and drawbacks. 

Although these scales are validated and standardized, they are, first, often depen- 
dent of an informant or a caregiver, which may include biases and, second, mainly 
performed in the control settings of the hospital. Being able to automatically assess 
ADLs at the home environment would benefit individual’s health in terms of disease 
prevention and treatment but would also enhance individual’s quality of life and 
independence. In the following section, we will review the current researches on the 
use of technologies to assess one’s ability to perform ADLs, mainly indoor-outdoor 
mobility and nutrition. 


Technology-Enabled Tools to Assess Indoor-Outdoor 
Mobility 


Over the last few years, Global Positioning System (GPS) technology, embedded in 
personal smartphone, has become the leading solution for outdoor positioning, and 
new technologies for indoor positioning and navigation exponentially expanded. 
Smart devices with embedded inertial sensors, radio beacons and image processing 
are just few examples of the systems deployed to assess indoor mobility. As most of 
the times, choosing one solution for any type of mobility assessment scenario is 
proven to be not possible, ongoing developments tend to combine different tech- 
nologies to find the best balance between precision, cost, robustness, scalability and 
energy consumption [31]. With respect to the ADLs, such solutions enable assess- 
ment of the indoor-outdoor mobility. Some of them may enable recognition of activ- 
ities such as transfer, use of stairs, dressing, meal preparation or use of rooms 
(bathroom, toilet and kitchen). Privacy concerns and environmental context recog- 
nition are often discussed in the research when choosing one or more systems for 
assessing mobility. In this section, we will review the current research on systems 
for mobility measurement by emphasizing two main categories: activity recognition 
and wireless indoor-outdoor positioning systems. 
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2 Assessing Activity of Daily Living through Technology-Enabled Tools: Mobility... 31 
Activity Recognition 


Activity recognition is focusing on distinguishing a person’s activities (sedentary: 
sleeping, sitting /active: walking, using stairs, cooking, etc.) in a given space is now 
currently performed using two types of technologies: vision-based systems and 
sensor-based systems [32, 33]. 

The vision-based recognition focuses on the processing and evaluation of video 
data from cameras or low-cost integrated depth sensor. Cameras installed in fixed 
positions can be fitted to the environment. In this case, the goal is to locate a moving 
target in images that are taken by one or more cameras [31]. Commons examples 
include video-based exercise program which allow user, caregiver or family to track 
daily exercise. Ayase et al. [34] investigated a “multimedia fitness exercise progress 
notes” which forwarded the video recording of elderly exercise movements to a 
research center, which were used to identify the duration, the distances and the 
angle of the movement. Another study [35] tried to analyze sit-to-stand motion from 
monocular videos by comparing the images received with the motion of a human 
body model. Li et al. [36] proposed a system that could extract gait features by using 
images from two orthogonal viewing cameras. Although optical systems show bet- 
ter accuracy nowadays [37], they remain intrusive, costly and are often limited to 
specific environments in which variable lighting and other disturbances can be con- 
trolled. However, attentions are now given to mobile and low-cost solutions based 
on camera embedded in mobile phone [38, 39]. Kahlert and Ehrhardt [40] used a 
photo-based ambulatory assessment to measure out-of-home mobility and social 
participation of older adults. The use of video and image may however be recog- 
nized as potentially privacy-threatening. 

Sensor-based recognition includes the use of pressure sensors, accelerometers, 
magnetic sensors and gyroscopes embedded within an environment, where the indi- 
vidual is active (or not). On one hand, smart floors and smart furniture are based on 
pressure sensitive sensors installed on or under the corresponding materials and 
focus either on load cell systems or pixelated surfaces. Load cell systems use sepa- 
rated sensors embedded on tiles and try to compute user location by sensing vertical 
force applied to the tiles [41, 42]. Pixelated surfaces on the contrary embed many 
sensors working as binary switches when someone is standing or moving on the 
surfaces [43-45]. Other furniture such as chairs and beds can also be embedded 
with pressure sensors to assess user mobility. Merilampi et al. [46] tested an exer- 
game coupled with a smart chair where users were asked to stand up and sit to play 
the game. On the other hand, the emergence and progress of wearables technologies 
have opened a door to develop different tools to assess individual’s mobility. Non- 
invasive off-the-shelf sensors such as wearables, smart clothes or smartphones have 
gained more and more attention for their potential to recognize indoor and outdoor 
activities while being continuously embedded on/around the user. Several reviews 
[47—49] discuss the potential of physical activity monitoring using accelerometry 
techniques to enable activity measurement of individuals in a free-living environ- 
ment. Steps and activity intensity are often measured through an off-the-shelf sen- 
sors or a smartphone and used to estimate energy expenditure following a physical 
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activity. Furthermore, Berglind et al. [50] showed in their study that smartphone 
apps can be as efficient as supervised exercise sessions (assessing the activities of 
the user) to improve body composition and cardiorespiratory fitness. 


Wireless Indoor-Outdoor Positioning Systems 


Through the past few years, several wireless technologies and techniques have been 
used and researched to assess indoor-outdoor positioning and navigation. Infrared, 
Ultrasound, Wi-Fi (Wireless Fidelity), RFID (Radio Frequency Identification), 
Bluetooth are mainly used indoor [31] while GPS (Global Positioning System) 
often represent outdoor solution [51]. 

All of them present advantages and drawbacks, which have to be considered 
while choosing the right solution. Infrared transmits infrared signals and compute 
the distance between the emitter and the receiver to locate the user. The main advan- 
tages are the absence of radio electromagnetic interference and an adjustable power 
of transmission. However, it can be costly to implement, less accurate and often 
requires a line of sight between transmitter and receiver to function properly [52]. 
Ultrasound systems use similar techniques by transmitting ultrasonic waves to 
assess the distance between fixed receivers and a mobile target. Synchronization of 
multiple receivers is then necessary. This system is relatively low-cost and could 
bypass indoors obstruction, but large-scale implementation remains complex and 
temperature variation could influence the speed of the sound [53]. Wi-Fi-based and 
Bluetooth-based systems are also known for being low-cost as it can localize every 
Wi-Fi compatible device without any extra installation, but it has lower accuracy and 
could induce large power consumption for smartphone [54, 55]. RFID system works 
without direct line of sight. Nevertheless, the intensity of the signal depends on the 
density of the obstacles in the building, and therefore precision is always reduced. 
However, RFID technology has more benefit such as better data rate, high security 
and compactness [56]. GPS-based applications are now widely used to track outdoor 
mobility. GPS-based navigation services are also often investigated in the context of 
facilitating visually impaired people mobility [57]. When combined with other 
devices such as accelerometers, the GPS system shows better accuracy in identify- 
ing the type of activity [58]. Weber and Porter [59] examined the feasibility of using 
GPS watch and accelerometers to monitor walking and community mobility in older 
adults and found promising results for monitoring community mobility patterns. 


Technology-Enabled Tools to Assess Nutrition 


Regarding techniques and technologies to assess nutrition which is important when 
assessing the meal preparation and feeding abilities of an individual, we also distin- 
guish between vision-based systems, sensor-based systems and self-assessment via 
mobile applications. 
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Ghali et al. [60] designed a system using cameras to provide real-time feedback 
to stroke patients while performing daily activities such as making a cup of coffee. 
They used histogram-based recognition methods to compare the recorded move- 
ment to a target task model and identify key events. P-W Lo et al. [61] investigated 
a vision-based dietary assessment approach to estimate food volume by using 
mobile phone 3D camera, deep learning and deep sensing techniques. They con- 
cluded an efficiency in portion size estimation even under view occlusion (food 
items scanned only from the front view). Pettitt et al. [62] combined micro-camera 
images with food diaries and improved the accuracy of dietary assessment. Their 
system also provided valuable information on macronutrient intake and eating rate. 
Gemming and al [63]. used a wearable camera to capture and categorize the envi- 
ronmental and social context of self-identified eating episodes and found that these 
contexts could be assessed objectively by using wearable cameras. Most of the eat- 
ing episodes were at home and indoors, seated at tables or sofas but also standing or 
at a desk. 

Sensor-based systems include the uses of furniture embedded sensors such as 
smart fork, smart table or smart plate. Zhou et al. [64] developed an unobtrusive 
smart table surface and were able to distinguish between different eating actions, 
such as cutting, scooping and poking. They could also indicate the type of food 
taken and the way the meal is consumed. Qualitative study [65] evaluating smart 
fork to decrease eating has been conducted and demonstrated user awareness of eat- 
ing rate. However, the incapacity of the fork to consider meal characteristics, were 
less appreciated. Huang et al. [66] prototyped a smart utensil which analyzes light 
spectra reflected by foods that differ for every food ingredient. They could recog- 
nize up to 20 types of aliments with 93% accuracy. Mertes et al. [67] proposed a 
novel plate system that can detect weight and location of individual bites during 
meals. They used a compartmented plate were filled with different type of food and 
initially weighed. Depending on the weight variation, they could detect the category 
of food eaten by the user. 

The most common techniques to assess nutrition are through self-assessment via 
an application or a platform. Todays’ systems are not only trying to measure food 
intake but based on nutrition habits, recommends healthy diet. Food recommender 
systems have received increasing attention to help people adopt healthier eating 
behavior. These systems focus on suggesting appropriate food items based on indi- 
viduals’ preferences and health conditions [68, 69]. Nevertheless, despite the exten- 
sive research and the large number of existing nutrition applications, food 
recommender systems are still facing many challenges in terms of nutrition habits 
tracking and delivery of the proper recommendations [70, 71]. People find it often 
difficult and time consuming to enter manually their food every day in the system. 
Many studies are also facing the uncertainty of the information given by the users as 
they may not know or tend to forget what they have eaten which makes it more chal- 
lenging for the system to recommend the correct food item. Additionally, many 
studies have shown that the recommendation is not necessarily followed by a change 
in the behavior [72]. There is still a lack of understanding in how to incorporate 
efficiently behavior change techniques into a food recommender system. 
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Use Case: Nutrition Assessment 


With the recent development in natural language understanding, conversational 
agents defined as a dialog system that supports human-like conversational interac- 
tion, have gained popularity over the past three years. In the nutrition domain, they 
allow to collect user data in an easy and user-friendly manner. Researchers have 
developed a Web-based prototype of a speech-controlled nutrition-logging system, 
which converts the entry spoken by the users into calories intake [73]. Others 
researched a chatbot that help people reduce their meat and increase fruits and veg- 
etables consumption [74]. Users were able to set nutrition goals themselves and had 
a follow-up with the system every day. Although, only 11% could reach their objec- 
tives, more than half of the participants showed positive changes in their nutri- 
tion habits. 

We developed a conversational agent called *MiranaBot" that helps the user to 
assess their nutritional patterns and to be aware of their eating habits in terms of 
variety and regularity. 

Interviews, observations and focus groups, presented in the methodology below, 
were conducted to collect the requirements from users and caregivers for an effi- 
cient nutritional conversational agent. 

Rather than focusing on food quantity and nutritional value, the system targets 
the variety of the individuals’ diet. From a regular description of the individual's 
meals during a certain period, *MiranaBot" is able to assess the quality of their 
nutrition, identify the foods they need to consume less, and explain why and how to 
replace them. The system proposes personalized solutions tailored to the older 
adults’ needs and context (Fig. 2.1). 


Fig. 2.1 Graphical User Interface—MiranaBot— Speech interface 
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We conducted a qualitative study with 10 obese and diabetic patients, 2 nutritionists 
and 1 physician with a specific expertise on therapeutic education of chronically-ill 
patients from University of Geneva hospital in Switzerland. Nutrition and overall 
self-management with respect to nutrition, usually implying the need for nutritional 
behavior change, plays a vital role in the self-management of disease for these 
patients; their future health state depends on the daily nutrition patterns. Observation 
sessions, focus group and semi-structured interviews were performed to get insights 
from the different participants. The prototype was developed through an iterative 
process, following the agile development principles [75]. A first prototype was 
developed within 4 weeks and changes were quickly implemented according to the 
participants feedback and in collaboration with them. 

To understand patient's current self-management with respect to nutrition, and 
identify barriers for adopting a healthy nutrition, we first observed 4 consultation 
sessions led by a physician at the therapeutic education center. In addition, we fol- 
lowed a group of 10 patients for one-week period, while they came to the center for 
a workshop to learn about their disease. To inquire about their needs and their issues, 
we conducted a focus group with the patients, while presenting them the first proto- 
type of the application. We conducted a semi-structured interview with one physi- 
cian and two nutritionists, who helped us to understand the real issues faced by the 
patients as well as the complexity of nutritional behavior change. The nutritionists 
participated in creating the nutritional database, we later used in the application. 

We then selected the corresponding Behavior Change Techniques (BCTs) from 
the literature aiming at and answering and fulfilling the needs and requirements 
identified for the MiranaBot. These BCTs are then converted into a set of function- 
alities that we incorporated in MiranaBot. 


Factors Influencing Nutrition Self-Management 


Our observation followed by the discussion with the patients and the health profes- 
sionals gave us insights on the barriers for patients to adopt a healthy nutrition as 
well as their needs and requirements for a successful behavior change. N1, N2 and 
P are used to refer to the nutritionists (N) and the physician (P). Patients are 
addressed as P1 to P10. 


Barriers to Healthy Nutrition 


Lack of regularity, lack of variety, false belief and hunger unawareness were identi- 
fied as the main barriers to healthy nutrition. 
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Lack of Regularity 

Nutritionists identified the lack of regularity in eating as the first issue commonly 
faced by their patients. N1 said “Most of our patients are not eating three times a 
day. They are not taking breakfast, neither a lunch, which make them starve for a big 
quantity of food later during the day or at night." P6 mentioned “I don't really have 
time because of my job, so I'm just grabbing some food that I can easily find when 
I finish my round. It's easier like that for me." P3 added "I don't eat breakfast, I'm 
used to that. At lunch, I just grab some sandwiches, but I come back home, I think I 
deserve a big dinner. I'm usually very hungry at night." 


Lack of Variety 

Health professionals also mentioned the lack of food variety as a recurring issue 
observed in chronically-ill patients. N1 stated "We often share the optimal plate 
during our workshop because most of our patients have difficulty to bring variety to 
their food. They are often eating the same aliment every day". P4 said “I have to 
admit that I'm always eating pasta and pizza, because I like them, and you know, it's 
not expensive.” P9 added “I’m always buying the same thing when I do my grocer- 
ies. I don't allow a lot of surprise about my food. I already have my personal habits." 


False Belief 

Following a consultation with a patient, the physician shared with us that one of the 
most difficult issue they face as health professionals is patients’ false beliefs. P said, 
"As one of my patients heard that olive oil was good for the heart, he started to put 
a big quantity of olive oil in everything he eats." During the session with her doctor, 
P2 shared "You know doctor, I only eat fresh cheese every day. Whatever is the 
menu, I always have my fresh cheese. It is fresh, so for me that’s healthy." 


Hunger Unawareness 

Nutritionists and physician both mentioned the complexity of nutritional behavior 
change due to the different external and internal factors influencing people to eat. P 
shared with us "Most of the time, when patients comes here, it is not only about 
food. It's about how they face the loss of their loved ones or the family problem they 
have. It's much more than a food addiction." N2 stated "It often happen that the 
patients do not eat because they feel hungry but rather to fill a void inside them." P7 
added "You know, I just eat automatically because there is food there, I don't think 
too much about why." P4 stated "I'm not always feeling good after eating, that's 
how I somehow realize maybe that was too much." 


Needs and Requirements 


Monitoring, education, empowerment and practicality were identified as the main 
needs and requirements and facilitators for nutrition behavior change approach. 


Monitoring 

Patients and health professionals both shared the need for a monitoring tool to raise 
self-awareness on patients' nutritional habits. P5 said "It's true that I don't always 
remember the food I ate. So, I don't really realize how I eat. If a tool can help me 
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know that, why not?” P7 added “The nutritionist asked me to describe what I ate on 
a diary, so that she can know how to help me. I guess it’s important to monitor what 
we eat but it is not always easy.” N2 stated “Just being aware of what and how they 
eat would already represents a big advancement because a lot of my patients do not 
even think of how much and why they eat. That’s the real issue. It is essential for 
them to monitor their eating habits.” 


Education 

The plethora of nutritional advice and beliefs accessible on the web make it often 
difficult for patients to identify trustworthy information. Patients appreciated the 
one-week workshop organized by the hospital as it taught them trustworthy and 
practical tips to change their habits. P6 said “You know before coming here, I 
thought drinking smoothie should be fine as it is just fruits. I did not know that fruits 
also contain sugar and drinking a lot of smoothie is really not healthy.” N1 men- 
tioned “During our session here, we teach our patients to always eat proteins, carbs 
and vegetable. We share simple tricks to measure portion. Our patients need that.” 


Empowerment 

From the perspective of the health professionals, there is no “one size fit all” solu- 
tion for everyone. Their role remains to give the necessary education. It is then up 
to the patients to build their own strategy and apply it for changing their behavior. 
P4 effectively confirmed “J won’t like it if you tell me what I have to eat. I should be 
free to choose what I like as long as I follow your general recommendation.” 


Practicality 

Most of our participants expressed the need for a practical and simple solution. P8 
shared “I used a nutrition application before to record my food, but it was too time 
consuming, so I gave up.” N2 mentioned “Specifying the quantity of your pasta and 
beef is not so obvious. Patients need a hands-on tool that would motivate them and 
make their life easy.” 


Summary 


Lack of regularity, lack of variety, false belief and hunger unawareness were identi- 
fied as the main barriers to healthy nutrition; whereas monitoring, education, 
empowerment and practicality were identified as the main needs and requirements 
for nutrition behavior change. Considering these findings, we selected appropriate 
BCTs to include inside our systems, as described in the next sections. 


MiranaBot: Behavior Change Theory and Techniques in Action 


Multiple researches have been conducted to identify motivational strategies for 
behavior change, especially in the nutrition and physical activity domain [76, 77]. 
Studies grounded in behavior theory appeared to show a positive impact on patient 
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Table 2.2 Mapping of requirements/barriers and the proposed BCTs 


Needs and Barriers to healthy 
requirements nutrition BCTs 
Monitoring Self-monitoring (eating habits history) 
Lack of regularity Personalized visual feedback (visual cue showing 
the time of eating) 
Lack of variety Personalized visual feedback (illustrating the type 
and the percentage of food eaten on plate) 
Education False belief Personalized education (through the chatbot) 
Empowerment Goal setting (with information on goal progression) 
Hunger unawareness | Self-awareness (asking the reason of eating before 
each meal) 
l Practicality Simple and user-friendly system (speech interface, 
visualization) 


health behavior. Sawesi et al. [78] demonstrated a significant relationship between 
theory-based health behavior change intervention and patient engagement. Various 
systematic reviews and meta-analysis [79-81] showed that efficacy of dietary inter- 
vention is associated with well-defined behavior change techniques like: social sup- 
port, increased contact frequency with the therapeutic team, goal setting, just-in-time 
feedback on behavior, review of goal progress and self-monitoring. 

There are many models and theories on behavior change, some focusing on parts 
of the behavior change process [82], while others are more holistic, aiming to 
include all factors that can influence behavior change [83]. “MiranaBot” is based on 
the Self-regulation theory [84] and the COM-B (Capability, Opportunity, Motivation, 
Behavior) model [82]. Self-monitoring, personalized visual feedback, goal setting, 
self-awareness and personalized education were the behavior change techniques 
incorporated in the tool. Table 2.2 summarizes the mapping of the different require- 
ments, barriers with the proposed BCTs (Fig. 2.2). 


Self-Monitoring 


“MiranaBot” allows the users to track their nutrition easily by asking them to briefly 
describe what they ate after each meal in terms of portions. The system will assess 
the portion size and initially train the users on the portion size to help them describe 
their meals as best as possible. 


Personalized Visual Feedback 
“MiranaBot” provides visual feedback every day regarding nutrition variety and 


regularity. Food variety is illustrated on a plate showing the ratio of proteins, carbo- 
hydrates and vegetables consumed by the user versus the optimal plate suggested by 
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Fig. 2.2 Behavior change theories and techniques operationalized within MiranaBot 

the Swiss Society of Nutrition [85]. A visual watch resuming the individual's eating 
times is intended to encourage self-reflection on their nutrition regularity. 

Goal Setting 

Following a training period analyzing the eating habits of the users, “MiranaBot” 
allows them to set personal goals, such as "eating three meals a day" or reducing 
"bread" and to define a timeline to reach their goals. Prompt reminders and useful 
information are provided to the users to help them reaching their goal. 


Self-Awareness 


Research has demonstrated that the reason to eat for most patients with chronic 
conditions are rather linked to psychological issues (emotions, stress, anxiety) or 
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automatic behavior, than hunger itself [86, 87]. “MiranaBot” raises awareness of 
real physical hunger by asking users how they are feeling before and after each meal. 


Personalized Education 


“MiranaBot” is developed to be continuously available to answer users’ questions 
during the process of change. Advice on food item alternatives, benefits of healthy 
nutrition behavior and promptly reminders towards their personal goals are fre- 
quently delivered. 


MiranaBot: Components and Functionalities 


“Mirana” refers to the overall system, providing ““MiranaBot” service, as one of the 
components. Figure 2.3 illustrates the process and the different technical modules 
of the conversational agent. “Mirana” is composed of four main components: the 
speech to text module, the text analysis module, the natural conversation module 
and the graphical user interface. We denote as “nuggets” the results of “Mirana” 
analysis after a certain period of data collection from the individual; focusing on 
types of foods that may need to be replaced by others, to achieve healthier nutri- 
tional outcomes. For example, a nugget could be “bread”, then the recommendation 
through the natural conversation module or the Graphic User Interface Module 
would be “let’s reduce bread consumption by 1⁄4 for a week. Here is what you can 
eat instead of bread". 


Speech to Text Module 


Users can easily describe, vocally, what they have eaten after each meal. This 
voice description is then converted in real time into text using DialogFlow, 
Google's human-computer interaction technologies based on natural language 


User Voice Input Speech-To-Text Text Analysis 


Natural Conversation 


& 
Graphical User Interface 


Fig. 2.3 Process and components of Mirana, including MiranaBot 
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conversations.! Data are stored in the secure server of the researcher’s institution 
in Switzerland. MiranaBot is now available in three languages, namely French, 
English and Danish. 


Text Analysis Module 


MiranaBot analyses users’ every day food data to provide them with daily and 
weekly visual feedbacks. MiranaBot also detects frequencies, varieties, and quanti- 
ties (in terms of portions) of a specific food (or category of food). With the help of 
nutritionists and dieticians, we were able to build a database of food classified by 
categories. We have set rules for each category, which are then used by our algo- 
rithm to identify recurring patterns (e.g., eat too much bread = more than the normal 
consumption defined by the rules). 


Natural Conversation Module 


One important functionality of “Mirana” is its ability to discuss, have a conversation 
and answer the questions from the user. As mentioned previously, the conversa- 
tional module is built upon DialogFlow developed by Google. Their module uses 
machine learning for small talk conversations to recognize user basic questions. In 
addition, we developed specific intents (or functions) related to our case to give 
“Mirana” the ability to answer questions about nutrition. 


Graphical User Interface 


To maximize the output provided to the user, MiranaBot uses the screen of the host 
device (phone, tablet, computer) to display graphical feedback in addition to the 
vocal answers. The regularity of the nutrition and the variety of the food eaten by 
the users are displayed on the screen to induce better self-reflection. While describ- 
ing their food, the user can also check, in real time, if what they said was transcribed 
correctly. In case of error, they have the possibility to edit the entry through the 
graphical user interface. The user interface is also used to share articles, tips or to 
send useful notifications to the user. 


'https://cloud.google.com/dialogflow 
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Conclusions 


This chapter reviews existing validated scales to measure ADLs and different 
technology-enabled tools, methodologies and strategies currently used to assess the 
ability to perform ADLs, namely mobility and nutrition. Although different scales 
and techniques exist, we found that there is no one-size-fit-all solution to assess 
ADLs. Depending on the individual’s conditions and environmental context, 
researchers and designers may choose one technology or even combine several of 
them to reach better accuracy. Each of the solutions has their own limitations. 
Despite the plethora of research in this area, most of the studies were conducted in 
controlled environment or in short time period. Further investigations are needed, in 
this promising area of Quality of Life Technologies (QoLT), specifically to 
assess ADLs. 

In this chapter, we also described the development of a conversational agent to 
promote healthy nutrition for chronically-ill patients. We conducted a qualitative 
study with 13 patients and health professionals who helped us identify the main bar- 
riers as well as the needs and requirements for adopting a healthy nutrition behavior. 
Lack of regularity, lack of variety, false belief and hunger unawareness were identi- 
fied as the main barriers to healthy nutrition; whereas monitoring, education, 
empowerment and practicality were identified as the main needs and requirements 
for nutrition behavior change. These findings allowed us to suggest the appropriate 
behavior change techniques to be leveraged in our systems, which are self- 
monitoring, personalized visual feedback, goal setting, self-awareness and person- 
alized education. Finally, these were translated into a set of functionalities that build 
up to construct our final solution: MiranaBot. We envision that solutions similar to 
MiranaBot will be leveraged in the near future to enable easy to use and highly 
engaging assessment and potential improvement of the Activities of Daily Life and 
behaviors, leading to better life quality of individuals at all ages. 

The QoLT we proposed here falls under the definition given by Wac [88], as a 
system for assessing or improving the individual’s QoL relying on a hardware and 
software technology via a visual and an auditory interface. The primary aim of our 
QoLT here was mainly to assess nutritional habits. However, it also holds the poten- 
tial to enhance individual’s nutrition. As an extension, QoLT can also potentially be 
tools using prediction from current behavior to prevent undesired future behavior. 
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Abbreviation 


AAL Ambient Assistive Living 
AHT Assistive Health Technology 
CPS Cyber-Physical Systems 

IC Integrated Circuit 

IMUs Inertial Measurement Units 
IoT Internet of Things 

NFC Near Field Communication 
RF Radio Frequency 

RFID Radio Frequency Identification 
RSSI Received Signal Strength Indicator 
TO Transmit Only 

UHF Ultra-High Frequency 

WSNs Wireless Sensor Networks 


Introduction 


Human lifespans will continue increasing as the average quality of life improves. 
Evidence of this can be seen in recent reports that highlight the significant increase 
in aging population, especially in developed countries [1—3]. As one would antici- 
pate, the global population of people aged 60 years and older will grow by 25096 in 
2050 as compared to 2013 [4]. Likewise, as society ages, long-term healthcare 
expenditures are projected to increase [5]. In order to maintain a healthy aging 
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population, the employment of Assistive Health Technology (AHT) increases [4]. 
Based on this, great efforts are being made towards achieving greater expectations 
of the quality in healthcare systems [3]. There is no doubt that rapid technological 
advances will revolutionize research in the twenty-first century in a number of dis- 
ciplines; namely human health. New approaches to monitor human health, behavior, 
and activity will be enabled. Medication adherence is an important component of 
health and well-being, with voluminous studies showing the importance of adequate 
medication adherence [6, 7]. 

Achieving healthy aging is challenging and thus requires several important strat- 
egies. Undoubtedly, correct medication is one of these strategies that are mainly 
related to the individual’s behavior. In addition, it is well-known that medications 
are the primary approach for treating most illnesses [8]. Hence, it requires the indi- 
vidual to take the medication as directed by the healthcare professional [9]. However, 
medication adherence remains a common issue within the healthcare sector, and 
especially among older adults. In fact, more than 50% of the older people are living 
with multiple chronic illnesses. Thus, routine monitoring and assessment of the 
individual’s adherence is crucial to improve their health outcomes [10]. To be suc- 
cessful, this should be performed using accurate assessment methods. Current 
assessment methods of medication adherence have advantages as well as 
limitations. 

With the aim of describing how the state-of-the-art technology on medication 
adherence monitoring can improve healthcare systems, we divide the present chap- 
ter into several sections based on the main monitoring or sensing technology used. 
We also compare the different medication adherence monitoring techniques and 
approaches related to accuracy, energy efficiency, and user’s comfort. Given the 
importance of technology embodiment in medication adherence systems, this chap- 
ter addresses the need of researchers and investigators of healthcare monitoring in 
both the engineering and medical societies. 


Background 


Medication Adherence 


Medication adherence can be defined as the extent to which a person-taking medica- 
tion adheres to a self-administered protocol [11]. In other words, medication adher- 
ence refers to the medication-intake behavior of the patient conforming to an agreed 
medication regimen specified by the healthcare provider with respect to timing, 
dosage, and frequency [12, 13]. From another point of view, non-adherence refers 
to the failure of taking medication as prescribed, including in-consistency, missing 
doses, and failing to re-fill the medication. Nonetheless, studies showed that failure 
to meet the medication-intake regime can result in emergence of drug resistance, 
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accelerated progression of disease, many irrevocable health complications [13, 14], 
and increased mortalities [15]. 

The benefits of adhering to medication regimens are many. However, for the 
patient, high adherence to prescribed medication leads to less health complications, 
more treatments’ benefits, and potentially active drug effect in the case of com- 
pletely treated infectious disease [12]. Another benefit is that medication adherence 
helps in minimizing drug wastage and reducing healthcare costs [16]. On the other 
side, poor medication adherence proven come with degradation in the health of the 
patient that may potentially lead to lower quality of life. 


Medication Adherence Monitoring 


Full adherence to medication is required as the drug can be effective only when it is 
taken in the proper dosage [12]. Nonetheless, maintaining strict medication adher- 
ence is required that deems maintaining administration timing, dosage quantity, and 
frequency [17]. A wealth of reports revealed that up to 50% of the patients either 
never fill their medication prescriptions or do not use the medication as prescribed 
to them in medication regimens [18]. Unfortunately, poor adherence is prevalent 
among populations with chronic illnesses [19], which leads to hospital admission. 
In the US alone, poor medication adherence results in more than 100,000 mortali- 
ties annually, as well as hundreds of billion dollars of healthcare spending every 
year [20, 21]. A number of approaches have been used for the aim of monitoring 
medication adherence because it has been shown that improving adherence to medi- 
cal therapy would substantially lead to both health and economic benefits. 

In general, two key factors should be considered when discussing medication 
adherence. The first factor is monitoring, which is alternatively referred to as assess- 
ment, quantification, measurement, or evaluation. Medication monitoring means 
using some methods for observing if the patient has taken the medication or not. 
Hence, the effectiveness of the monitoring method plays a central role. The second 
factor is intervention. Intervention refers to the means that can be used for improv- 
ing adherence to medication or correcting it once erroneous or drift is detected. 
How- ever, the latter is more in the domain of the psychological and social sciences 
as it requires understanding the cultural, psychological and social factors that affect 
the patient’s behavior [22], and thus it is out of the scope of this chapter. 

Methods that have been utilized for measuring medication adherence so far can 
be broadly divided into two categories, direct and indirect [23]. Direct methods of 
measurement of adherence include direct observation of the patient while taking the 
medication, laboratory detection of the drug in the biologic fluid of the patient (1.e., 
blood or urine), laboratory detection of the presence of nontoxic markers added to 
the medication in the biologic fluid of the patient, and laboratory detection of the 
presence of biomarkers in the dried blood spots [24]. Meanwhile, the patient’s 
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self-reporting, pill-counting, assessing pharmacy refill rates, and using electronic 
medication event tracking systems are examples of indirect methods of measuring 
adherence. There is not a gold standard measurement system that fulfills the criteria 
for an optimal medication adherence monitoring. Each category comes with bene- 
fits and limitations. Direct measures are accurate, but they are invasive and expen- 
sive. In comparison, indirect methods are less expensive and provide good estimation 
of the medication adherence. However, these methods relay on the reliability of the 
user [25]. As such, these factors should be taken into consideration when selecting 
the adherence measurement methodology. 


Why Technology-Based Solutions? 


The development of Cyber-Physical Systems (CPS) that integrate computation and 
physical processes for healthcare, are advancing rapidly [26]. More recently, such 
systems included few sensing and monitoring devices associated with mobile 
devices such as smart pill bottles, smart watches, smart phones, and wearables. The 
combination of these smart monitoring devices with interventions that remind the 
patient in case a deviation is detected has proven to improve medication adherence 
[27, 28]. Compared to manual approaches, electronic-based approaches can reduce 
the cost and effort from the user’s interest. In addition, the accuracy of adherence 
measure, which is of great importance from the healthcare provider’s point of view 
can be enhanced when using electronic-based systems. Furthermore, as we live in 
the era of the Internet of Things (IoT) [29], where everything is connected to the 
Internet, a connected health paradigm is becoming a more dominant field [30]. One 
expectation of connected health is the automated capability of communicating the 
collected adherence measurements to the provider, and the feature of issuing 
reminder and alert messages based on the processed information [31]. Moreover, 
electronic measurement systems can be portable and thus provide timely and long- 
term monitoring without restricting the user's mobility. In spite of the fact that 
electronic-based modalities can outperform traditional ones, the majority of 
electronic-based approaches come with limitations that act as burdens on the users, 
as we will see in Sect. 3.5. In fact, some of them have not achieved much success due 
to these burdens [23]. Based on this, we conclude that there is no optimal electronic- 
based solution for medication adherence evaluation and, for that, much additional 
efforts will be required to realize accurate, low cost electronic adherence monitoring. 


Related Work 


In the past, a wide number of review studies that addressed the medication adher- 
ence problem have been created. However, most reviews studied the medication 
adherence from a clinical point of view along with interventions [6, 7, 11]. Moreover, 
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only a few studies have presented the electronic-based interventions [18, 23, 25, 32, 
33]. Little attention has been paid towards employing technology in medication 
adherence monitoring and enhancement as compared to the traditional modalities. 
These reviews have elucidated the role of technology-based solutions for medica- 
tion adherence assessment, the potential benefits and limitations, but, no detailed 
discussion on the cyber-physical system, including system design, hardware devel- 
opment, and data analytic of these solutions were given. 

A rare number of studies describe technology-based interventions for adherence 
monitoring and enhancement. For example, Park et al. [33] presented an overview 
of a number of electronic systems and methods of medication measurement. Other 
review articles have discussed the smartphones’ applications, and tablet applica- 
tions technology [25] for medication adherence that are in the form of automated 
reminder systems. In [34], some technological medication reminder approaches 
have been briefly described. It is worth mentioning that only a recent study by Rokni 
et al. [35] has reported some commercially available technology-based solutions. In 
addition, they provided a brief discussion of some clinical studies that involved 
electronic medication monitoring. It also discussed the challenges associated with 
medication monitoring technologies from data analytics, reliability, and scalability 
sides. It is obvious that these survey studies are limited in providing a detailed dis- 
cussion of the technical sides of the different technology-based sensing or monitor- 
ing approaches for medication adherence. 

The main objective of this chapter is to explore this topic further by taking 
account of other medication monitoring systems such as ingestible biosensors, and 
discussing the trade-offs of each technology in multiple dimensions. 


A Review of Medication Adherence Monitoring Systems 


Medication non-adherence is an extensively studied complex problem. The com- 
mon conclusion of these studies is that several interventions are required to improve 
medication adherence [18, 36]. Nonetheless, technological interventions are 
believed to be supportive tools in improving adherence. This is due to the fact that 
they allow timely monitoring, and generate useful information about the patient’s 
behavior for the healthcare provider. To date, a considerable number of systems 
have been proposed and developed that utilize monitoring and tracking techniques 
in various health-related projects, including medication adherence monitoring. In 
this section, we categorize and review the existing approaches on designing moni- 
toring systems for medication adherence applications using emerging 
technologies. 

Our review includes articles from journals, and conference papers and proceed- 
ings. We excluded articles classified as editorials, book reviews, white papers, or 
newspaper reports. While searching for papers, electronic databases including 
Google Scholar, IEEE Xplore, ACM Digital Library, Springer Link, MDPI, and 
Science Direct, were used. The descriptors we used were “medication adherence”, 
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or “medication intake", or “medication monitoring", or “medication compliance” in 
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combination with at least one of others, including “technology”, “sensor”, “smart- 
watch”, “wearable”, “smart bottle", “pill bottle", “pillbox”, “vision system", “Radio 
Frequency Identification (RFID)" and “Near Field Communication (NFC)". The 
search was inclusive of all years from 2004 through 2019. 

Using primarily the full text and the abstracts, we selected articles discussing 
medication adherence monitoring technologies and excluded papers discussing 
intervention applications. The literature review approach used in this paper follows 
an iterative and incremental procedure [37], and hence found and included new 
studies about medication adherence monitoring technologies and approaches to the 
surveyed studies. 

Table 3.1 provides a taxonomy of the approaches reviewed in this chapter. 
Table 3.2 summarizes the key properties of existing technology-based systems 
reviewed in this chapter. 


Table 3.1 A taxonomy of the technology-based approaches for medication adherence monitoring 


Main Secondary Monitored Activities and/ 
Reference Category Technology | Technology | or Subjects 
Hayes et al, (2009) | Sensor Smart pill = Lid opening 
[9] systems container 
Aldeer et al., (2019) | Sensor Smart pill - Lid opening and closure, 
[38] systems container bottle picking and flipping/ 
shaking 
Lee and Dey, (2015) | Sensor Smart pill - Lid opening and closure, 
[39] systems container box manipulation 
Kalantraian et al., Sensor Wearable Smart pill Pill bottle pick up and pill 
(2016) [40] systems Sensors container swallowing 
Wu et al., (2015) Sensor Wearable Ingestible Pill swallowing 
[41] systems sensors biosensors 
Kalantraian et al., Sensor Wearable = Pill bottle opening, pill 
(2015) [42] systems sensors removal, pill pouring into 
the secondary hands, water 
bottle handling 
Hezarjaribi et al., Sensor Wearable = Hand-to-mouth motion 
(2016) [43] systems sensors 
Wang et al., (2014) | Sensor Wearable - Taking a pill, drinking 
[44] systems sensors water and wiping mouth 
Chen et al., (2014) Sensor Wearable = Cap twisting and 
[45] systems sensors hand-to-mouth actions 
Mondol et al., (2016) | Sensor Wearable = User’s response in the 
[46] systems sensors form of voice commands 


3 Monitoring Technologies for Quantifying Medication Adherence 


Table 3.1 (continued) 


55 


Main Secondary Monitored Activities and/ 
Reference Category Technology | Technology | or Subjects 
Hafezi et al., (2015) | Sensor Ingestible = Medication ingestion 
[47] systems biosensors 
Chai et al., (2016) Sensor Ingestible - Medication ingestion 
[14] systems biosensors 
Agarawala et al., Proximity- | RFID = Pill bottle pick up 
(2004) [48] based 
systems 
Becker et al., (2009) | Proximity- RFID - Pill removal 
[49] based 
systems 
Morak et al., (2012) | Proximity- NFC - Pill removal 
[50] based 
systems 
Batz et al., (2005) Proximity- | Computer = Pill bottle opening, hand 
[51] based vision over mouth motion, bottle 
systems closing 
Valin et al., (2006) Vision-based | Computer = Pill bottle opening, pill 
[52] systems vision picking, pill swallowing, 
bottle closing 
Dauphin and Vision-based | Computer - Pill bottle picking, 
Khanfir, (2011) [53] | systems vision drinking a glass of water, 
putting glass back 
Huynh et al., (2009) | Vision-based | Computer - Tracking the face, the 
[54] systems vision mouth, the hands, and the 
medication bottle 
Sohn et al., (2015) Vision-based | Computer = Bottle weight 
[55] systems vision 
Li et al., (2014) [56] | Fusion-based | RFID Sensor Pill removal, hand motion 
systems networks 
Hasanuzzaman et al., | Fusion-based | RFID Computer Pill bottle removal, 
(2013) [57] systems vision tracking hands and 
medication bottle 
Suzuki and Fusion-based | Computer Sensor Pill bottle removal, user 
Nakauchi, (2011) systems vision networks behavior prediction 
[58] 
Abbey et al., (2012) | Fusion-based | Smart pill Mobile Pill removal 
[59] systems container application 
Boonnuddar and Fusion-based | Smart pill Mobile Bottle weight 
Wuttidittachotti, systems container application 


(2017) [60] 
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Table 3.2 Summary of main applications, strengths, and limitations of the current technologies 
used in medication adherence 


Category 


Main Application 
Differences 


Strengths 


Limitations 


Sensor | Smart Pill | Detects cap opening and | Possibility to System’s life is 
Systems | Container | bottle pick up allow mobility constrained by the 
Non-invasive battery Detect 
medication taking 
activity with low 
accuracy 
Wearable | Detects motions related | Possibility to User's comfort and 
Sensors to cap twisting, detect medication | social acceptance due 
hand-to-mouth, pouring | intake activity to their possible 
pill into the hand, and with sign accuracy | invasiveness 
pill swallowing Relatively easy to | Require frequent 
use Allow battery charging or 
mobility replacement 
Ingestible | Detect pill ingestion Possibility to User’s comfort and 
Biosensors detect concurrent | social acceptance 
pills ingestion System’s lifetime is 
Allow mobility constrained by the 
battery 
Security issues due to 
their limited resources 


Proximity-Based 
Systems 


Detects medication 
presence or absence 
within the proximity of 
reader’s antenna 


Non-invasive 


Need to be coupled 
with other monitoring 
or sensing techniques 
for verification 


Vision-Based Systems 


Detects medication 
presence or absence 
within the scope of the 
camera 


Non-invasive 


with tech or sensing 
techniques for 
verification 


 Fusion-Based Systems 


Try to verify the 
operation of monitoring 
the medication taking 
activity 


Higher accuracy 
as compared to 
standalone 
technology 


Resource consuming 
Do not usually support 


Need to be coupled 
mobility 


Sensor-Based Systems 


Recent years have seen the size, cost, and energy consumption of small wireless 
sensors decrease by several orders of magnitude [61]. Indeed, today, low-power 
wireless sensors can be bought for an affordable price. In the context of human 
health, sensor systems allow us to collect data on daily activities in a free-living 
environment and possibly over long time periods, seamlessly [62]. One promising 
application in that field is the monitoring and assessment of subject for medication 
intake [63]. In fact, sensor-based approaches are the most widely used among other 
approaches these days for adherence monitoring. Utilizing sensor networks into 
medicine intake and adherence monitoring systems comes with features and 
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benefits. The regularity in measurements, remote monitoring capability, and context 
awareness are a few examples [63]. In general, wireless sensors in this area of moni- 
toring can be put into two main categories based on the form of deployment: fixed 
and wearable. Fixed sensors are tied to minimally mobile objects such as pillboxes 
or pill bottles, and home apparatuses. Meanwhile, wearable sensors are lightweight, 
have high data fidelity, and mobile devices that are attached to the user’s body. In 
vivo or intra body communication and networking [64] is another emerging sensor- 
based communication and network technology within the IoT family, which is 
enabling a new set of healthcare applications. 

In this part, we describe the recent work on medication adherence monitoring 
using different forms of wireless sensing. 


Smart Pill Containers 


Pillboxes and pill bottles equipped with sensors have been developed for monitoring 
the medication-taking activity. In this context, Hayes et al. [9] developed 
MedTracker(Fig. 3.1). It is one of the earliest approaches that uses a 7-day multi- 
compartment pillbox embedding plungers in each compartment. It was designed to 
detect the lids of boxes opening as the plungers would activate a switch inside the 
pillbox that then triggers the micro-controller. The system uses Bluetooth technol- 
ogy for wireless transmission of the data to a nearby computer. Data was transmit- 
ted over the Bluetooth link every two hours for the aim of prolonging the lifetime of 
system, which was using a 9 V battery. The system includes RAM for storing medi- 
cation taking events when there is no connection with the base station. However, it 
is obvi- ous that the system is simple and is error prone as it considers any lid open- 
ing event as medication taking. Regardless of its simplicity, the system achieved a 
lifetime of eight weeks only, given it was powered from a considerably big battery. 

For a project that intended observing daily living of elderly people, Lee and Dey 
[39] developed a pillbox similar to that reported in [9]. A 7-day compartment has 
been equipped with a Microcontroller (MCU), a ZigBee wireless module, an accel- 
erometer, and a battery (Fig. 3.2). Data were transmitted to a nearby laptop for fur- 
ther processing. The aim of this system was for human-computer interaction studies. 


Fig. 3.1 MedTracker 
prototype pre- sented in [9] 
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Fig. 3.2 The system 
developed by Lee and 
Dey [39] 


Fig. 3.3 The system prototype developed by Aldeer et al. [66]. (a) Pill bottle. (b) Bottle compart- 
ment and cap with the sensors shown 


In another approach that was recently carried on by Aldeer et al. [65, 66], a smart 
pill bottle and a sensing framework for medication adherence monitoring have been 
proposed. As shown in Fig. 3.3, they built a 3D printed pill bottle equipped with a 
magnetic switch sensor, an accelerometer, and a load cell. Furthermore, the system 
uses PIP-Tag mote [67] as a platform for collecting the data from the employed sen- 
sors and then transmitting them wirelessly to a base station attached to a nearby 
computer. 

Such an approach aims to eliminate the intervention and attachment of sensors to 
the human body, and by that it ensures user's comfort while maintaining accuracy 
by using the accelerometer sensor. However, the system does not ascertain if a pill 
is ingested or not by the user. 


Wearable Sensors 
In the recent years, Inertial Measurement Units (IMUs) have seen rapid achieve- 


ments from both the cost and intelligence points of view [68]. IMUs usually consist 
of accelerometers, gyroscopes, and magnetometers, or a combination of these [69]. 
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They have been widely used in healthcare applications by sensing motion and 
track- ing individuals [70]. Ultimately, the usage of motion sensors can help in 
revealing possible information about individual’s health [66]. In this part, we pres- 
ent many wearable sensing systems and place them in two categories, depending on 
the place- ment location of the body, neck-worn and wrist-worn. 

Neck-Worn Sensors: In one of the studies [40], the authors propose a wearable 
system for detecting user adherence up to the level of determining if the medication 
has been ingested. As shown in Fig. 3.4, they built a pendant-style necklace that 
includes a piezoelectric sensor, a Radio Frequency (RF) board, and battery. The 
piezoelectric sensor is used for sensing the mechanical stress resulting from skin 
motion during pill swallowing and generating voltage as a response. Major chal- 
lenges associated with this approach pertain to user comfort and social acceptance 
[71] as the necklace needs to be worn by the patient and placed in contact with the 
skin during dose swallowing. 
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Fig. 3.4 The neck-worn system pre- sented in [40] 
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Fig. 3.5 The system 
developed by Wu et al. [41] 


Another tool for assessing medication intake is using acoustic sensors in the 
form of neck wearables. Such an approach has been utilized for food intake moni- 
toring applications [72]. Although this approach requires further research, it shows 
promise for being applicable to medication monitoring [73]. Only one prototype of 
this class of wearables was developed by Wu et al. [41]. The neckwear device con- 
tains microphones, a flex sensor, and an RFID reader (see Fig. 3.5). The micro- 
phones and the flex sensor are to be employed for sensing throat movement and 
chewing sound associated with medication swallowing activity. However, the study 
did not include any validation trials, thus making it difficult to make conclusions 
about the performance, social acceptance, and comfort of this approach. 

Given the promise of acoustic sensing in food monitoring, it is highly likely that 
this technology will face the same challenges associated with other neck-worn sen- 
sors when applied in promoting medical compliance in older users [74]. 

Wrist-Worn Sensors: When reviewing sensor-based systems, one should not 
ignore personal sensors. Personal sensors are a class of wearables that can be used 
for fashion and tracking purposes, such as smartwatches [75]. Nonetheless, these 
wearables embed miniaturized and continuously progressing capabilities including 
Inertial Measurements Units (IMUs) (accelerometer, gyroscope, and magnetometer 
or a combination of these) [76, 77]. Thus, wearable and personal sensors have been 
recently used in many healthcare monitoring studies, including medication intake 
detection. The reason behind using IMUs in such systems is their ability to accu- 
rately recognize the intensity, direction, and angle of movements conjugated with 
medication intake activity in a 3D coordinate system [78]. Collecting such data will 
help in modeling the user's physical activity and then infer if it is associated with 
medication taking activity or not. In [43], accelerometer and gyroscope sensors 
embedded in a pair of smartwatches placed on both wrists of the user were used to 
sense and transmit readings associated with pill taking activity from 10 users. Using 
a decision tree classifier, the system was able to detect the wrist movement while 
taking medication with 78.3% accuracy using one smartwatch placed on either of 
the wrists. Moreover, the accuracy of the system was 86.2% when using two smart- 
watches for tracking the motion of both hands. 

Wang et al. [44] used accelerometery data samples from wrist-watches and 
dynamic time warping technique to test if a sample belongs to either activities: tak- 
ing a pill with water or drinking water and wiping mouth. Data from 25 individuals 
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were used to classify the hand movement gestures associated with one of the previ- 
ously mentioned activities. The system achieved 84.17% true positive rate. A fur- 
ther re- search study of Chen et al. featuring wearable sensors presents a system for 
detecting two actions “cap twisting" and “hand-to-mouth” from a triaxial acceler- 
ometer and a gyroscope [45]. Classification accuracies were 9596 and 97.596 for cap 
twisting and hand-to-mouth actions, respectively. 

Finally, termed MedRem, was presented in [46]. Unlike other approaches that 
used IMUs available on smartwatches, MedRem uses the speaker microphone on a 
smartwatch to provide reminders and track medication adherence via voice com- 
mands. When reminders are provided in the form of voice commands, it is expected 
that the user send a recording via the microphone sensor to confirm or postpone 
taking medication. The smartwatch then uses an android speech recognizer to ana- 
lyze user's input and update a server. The capability of recognizing native and non- 
native English speakers’ commands was 6.43% and 20.9% error rates. 

Advantages of wearable sensors approaches include the ability of monitoring the 
user behavior in a free-living environment [72]. Another advantage is the accuracy 
of sensor-based systems. However, a main disadvantage that is pertained with wear- 
able- based systems is the user acceptance and comfort, especially when consider- 
ing old people [71]. This is due to the requirement that the sensor should be attached 
to the user for possibly a long time and recharged frequently, as wearables are usu- 
ally powered by small batteries. 


Ingestible Biosensors 


The use of biosensors in connected health is in its infancy. However, with the intro- 
duction of In vivo communications, it can be expected that the biosensor technology 
will dramatically improve over time and increase in value to advancing healthcare 
delivery [64]. Ingestible devices are miniature capsule-looking devices that are 
digested and swallowed when taken through mouth like solid medications. These 
devices travel through the gastrointestinal tract and digestive system and collect 
data about specific physiological parameters [79]. One application of these devices 
can be for adherence monitoring, where data about drug consumption are collected 
and transmitted to a body-worn or nearby device for further post-processing [80]. 

Researchers from Proteus Digital Health, Inc. (Redwood City, CA, USA) have 
designed a micro biosensor that is intended to be integrated with pharmaceutical 
oral dose (pill or capsule) for evaluating medication ingestion [47, 81]. The sensor 
is built from an Integrated Circuit (IC) made of specific materials (including gold), 
with a food particle size. Upon contact with the gastric fluid, the ingestible sensor 
communicates with a wearable receiver worn by the patient and transmits a unique 
code. A mobile phone user interface can then identify the ingested medication based 
on the received code from the ingested biosensor. The designed device has been 
tested via multiple clinical studies. Furthermore, 412 subjects were involved in the 
clinical studies where they have performed more than 20,000 ingestions spanning 
5656 days in total. The detection accuracy was more than 99%. 
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MyTMed is another system that is based on ingestible biosensors [14]. The cen- 
tral part of MyTMed is the digital capsule that can encapsulate oral medication. It is 
made of a standard gelatin pill capsule that includes a sesame seed size RFID tag. 
Upon ingestion by the patient, the gelatin capsule dissolves in the stomach and 
releases the medicine along with the RFID tag. The electro-chemical reaction 
between the tag’s electrolytes in gastric acid forms a bio-galvanic battery that 
enables it to emit a unique code in the forms of packets to a body worn receiver. 
Eventually, the receiver utilizes short messaging service (SMS) to relay the packets 
to a cloud server that can be accessed by the caregiver. Based on a 10 participants 
trail study with 96 ingestion events, the system’s detection accuracy was 87.3% [82]. 

Advantages of biosensor-based techniques include their ability to detect concur- 
rent medication ingestion events with relatively high accuracy and no computa- 
tional cost. However, as such systems require external receivers to be adhered to the 
individual’s body, many users would object to wearing a banded device throughout 
the day and possibly for years (when considering people with chronic illnesses). 
Security and privacy are also an issue, with resource-constraint tags requiring low- 
energy and lightweight computing cryptographic tools [83]. 


Proximity Sensing 


The visionary concept of IoT relays on some technologies, among which is the 
proximity detection [84]. Hence, objects usage in our daily life can be monitored by 
sensing their proximity to other things. Two important wireless communication 
technologies that are currently used for proximity detection and sensing are RFID 
[85] and NFC [86]. Overall, RFID and NFC are contactless short-range communi- 
cation technologies that can be integrated in everyday life objects to sense the daily 
activities [87]. Here, we describe the RFID-based and NFC-based systems and their 
usefulness and shortcomings. 

An early demonstration that applied RFID technology was designed by Agarawala 
et al. [48]. The system uses an RFID tag attached to a pill bottle that is placed on a 
platform embedding an RFID reader and LEDs (Fig. 3.6). The LEDs flash to notify 
the patient when it is time to take medication. Using this system, it is inferred that 
the medication is taken when the medication bottle is picked from the platform and 
it is not within the coverage radius of the RFID reader anymore. The caregiver can 
track the patient's adherence via an Ethernet connection with the platform. Another 
RFID- based system is SmartDrawer [49], Fig. 3.7. A drawer with an RFID reader 
that is capable of inventorying the pill bottles that are stored inside it as well as keep- 
ing a record of drug taking activities, is used. The pill bottles are equipped with 
RFID tags for identification and tracking. The system records the type of bottle and 
when it is removed from the drawer. In other words, it is assumed that the medica- 
tion is taken when the bottle of that medicine is removed from the drawer and it is 
not within the scope of the RFID reader. Other short communications-based 
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Fig. 3.6 The RFID-based system devel- oped by Agarawala et al. [48] 


Fig.3.7 SmartDrawer system developed by Becker et al. [49] 
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approaches designed a smart blister that is equipped with a uC along with the NFC 
technology available on mobile phones, to develop an adherence tele-monitoring 
system [50]. The idea is that the smart blister records the event of pill removal and 
reports this activity to a mobile phone that is in the proximity via NFC. The mobile 
phone then communicates this event to a remote server to be accessed by the care- 
giver that assesses the medication intake adherence. 

Proximity sensing-based systems have advantages as well as limitations. The 
main advantage is the possibility of retrieving information such as dosage instruc- 
tions that may include timing, frequency, and quantity. Such information can be 
helpful when considering elderly patients. Another advantage is the non- 
invasiveness, as sensing tags are usually attached to the pill containers. However, 
the main limitation of these systems is the requirement that the pill container being 
located within a short distance (several centimeters) of the vicinity of the main part 
of the system, which is the reader. Most importantly, there have been some studies 
that addressed possible harm to the fetus that are associated with the exposure to 
Ultra-High Frequency (UHF) RFID readers during pregnancy [88]. 


Vision-Based Systems 


Recently, research in computer vision and image processing has attracted much 
attention, leading to the development of many algorithms for human activity repre- 
sentation and classification [89]. So far, vision-based systems have been the basis 
for a number of important healthcare applications. In the context of human activity 
recognition within smart environments or “Smart Homes” [75], where Ambient 
Assistive Living (AAL) technologies [2] exist; one choice for monitoring medica- 
tion intake is to use vision modules for identifying and tracking inhabitants, motion, 
gestures, and subjects. In this section, we depict the current vision-based systems 
for medication intake monitoring and discuss their pros and cons. 

In [51], a computer vision system was proposed for monitoring medication hab- 
its. The system uses one camera installed in the medication area, which may include 
a group of medication bottles (Fig. 3.8). The aim of this system was to track if the 


Fig. 3.8 The vision-based system devel- oped by [51] 
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right medication is being taken by the user. In order for the system to work, it is 
required that only one user appears closely in the field of view of the camera during 
the medication taking session. Algorithms for skin color distinction have been used 
in order to distinguish between skin and non-skin colors. First, the systems extract 
all skin regions of the person in front of the camera. Then, this information is used 
for detecting hand/face occlusions and hand/hand occlusions. Researchers used four 
users in different environments to evaluate the system. Another computer vision 
system for monitoring medication intake was developed by Valin et al. [52]. The 
system considered multi-state scenarios including bottle opening, pill picking, pill 
swallowing, and bottle closing. It uses color classification algorithms for person 
detection and motion tracking by distinguishing the person’s skin. In addition, col- 
ored bottles have been used for medication bottle detection. The recognition results 
were 90% classification accuracy for scenarios that differ from each other in the 
sequence of activities associated with medication taking. 

The work in [53] focused on developing a technique for background suppression 
of videos captured by low resolution cameras. However, the technique was only 
tested with one participant and no accuracy measurements were reported. 
Furthermore, the system’s accuracy may get affected for different colored clothes 
worn by the participants, as the experiments have been conducted with a participant 
wearing dark colors compared to the background. Another similar vision-based sys- 
tem developed by Huynh et al. [54] used a multi-level approach for detecting and 
tracking mobile objects during medication intake. The face, the mouth, the hands, a 
glass of water, and the medication bottle were tracked in this system. To achieve 
this, detection and tracking techniques for background subtraction, skin regions’ 
segmentation, and using color information for bottle detection are used. The average 
success rate of activity recognition was 98% from a population of three subjects. In 
later work, the authors directly use two cameras for the aim of occlusion handling. 

The literature also shows a monitoring system that consists of a digital scale and 
a camera that was presented in [55]. As illustrated in Fig. 3.9, a digital scale has 
been used such that it continuously measures and displays the medication bottle 
weight. The camera has been used to capture and send the scale’s readings displayed 


Fig. 3.9 The system developed by Sohn et al. [55] 
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on the screen to a nearby computer. Upon receiving the images, the computer then 
runs an image processing algorithm for processing the bottle’s weight. From the 
bottle’s weight decrease trend, the system can generate an alarm to remind the 
patient to take medication. It should be noted that although this work concentrates 
on vision analysis, it does not include any human subject tracking. It is obvious that 
such a system does not support mobility due to the fact that it requires the medica- 
tion bottle to always be placed on the weight scale, and thus provides only a limited 
view. Although vision-based systems will play an important role in AAL environ- 
ments, the main disadvantages of these approaches are their limitation in use and 
accuracy. In addition, these approaches may demand several resources, which can 
be expensive. Furthermore, as we progress further into the twenty-first century, 
users prefer fully mobile devices [90]. However, in contrast, vision-based approaches 
do not support mobility. 

Finally, another limitation is that the user is required to be within the scope of 
the camera. 


Fusion-Based Systems 


It is seen from the studies we covered that each approach comes with drawbacks. As 
such, fusion-based systems have been developed that aim at blending advances 
available from multiple techniques for enhancing one or more technical drawback 
[72]. In this section, we subdivide fusion-based systems into several categories, 
based on the blend of techniques used. 


Proximity-Sensor Systems 


In [56], Li et al. have designed a system that was built with a cylindrically shaped 
7-compartment pillbox, a wristband device, and a computer that all communicate 
with each other wirelessly. Fig. 3.10 shows the system. The pillbox is comprised of 
an Arduino MCU, a motor, a ZigBee transceiver, and an RFID reader. In addition, 
each compartment is embedded with a diode and a photo diode for detecting pill 
removal. The MCU controls the motor such that it rotates the compartment towards 
the user when it is time to take medication and when the RFID-based wristband is 
detected in the proximity of the pillbox. The wristband embeds an RFID tag, and an 
LED, and is used for collecting motion data associated with pill picking and taking. 


Proximity-Visual Systems 


A blend of RFID sensors and video camera has been used in [57] to characterize the 
medication taking activity in an in-home environment. In this work, medication 
bottles were equipped with RFID tags and stored in a medicine cabinet that embeds 
an RFID reader. The RFID technology is employed for identification purposes of 


3 Monitoring Technologies for Quantifying Medication Adherence 67 


Pillbox > iaaa ec ee 
$ - 


' 
ZigBee E 


Fig. 3.10 The fusion-based system developed by [56] 


the medication bottles placed in the cabinet. However, once a bottle is removed 
from the medication cabinet and it is out of the coverage of the reader's antenna, the 
identification process using RFID technology can not be achieved anymore. As 
such, the vision system is used such that it is activated once the medication bottle 
moves out of the range of the reader. The camera is used for tracking and verifying 
the occurrence of medication taking based on moving object detection and color 
model of the bottle. 


Visual-Sensor Systems 


Assistive living techniques have been used to track medication intake based on the 
patient's activity. One example is iMEC (Fig. 3.11), that has been developed by 
Suzuki and Nakauchi [58] for medicine timing and pill taking detection. Some 
home appliances (refrigerator, microwave oven, chair, and bed) have been attached 
with ubiquitous sensors for predicting the behavior of the patient. A medicine case 
equipped with a camera has been used for detecting pill removal. Eventually, the 
blend of data from these devices were used for confirming medication adherence. 


Sensor-App Systems 


Personal mobile device technology has witnessed a rapid progression in recent 
years. The services brought by mobile devices, such as the different means of com- 
munications and user applications, have enabled a host of possibilities. Thus, mobile 
applications’ industry has been in race, including those for promoting healthcare of 
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Fig. 3.11 iMEC system prototype devel- oped by Suzuki and Nakauchi [58] 


older patients [25]. Specifically, many mobile and tablet-based applications have 
been developed in the form of automated reminder systems [91]. 

In this context, the sensor-app approach blends the use of sensor networks and 
mobile-app approaches for medication adherence tracking and monitoring. Abbey 
et al. [59] developed a pillbox containing multiple compartments with ambient light 
sensor fixed in each of them and a WiFi connection. Also, a mobile app has been 
developed that contains the medicine schedule. The pillbox and the mobile app are 
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interconnected through an online data source. Hence, the mobile app generates 
alarms when it is the time of medication until the patient takes the medication from 
the pillbox or chooses to delay the action. In a recent study, Boonnuddar and 
Wuttidittachotti [60] proposed a pillbox-based system that uses the Arduino UNO 
WiFi and a load cell. Medication weight changes were reported to a server via the 
Internet. Also, a mobile application was developed that tracks the change in weight 
measurements and alerts the patient to take medication. The system was tested for 
160 times of medication taking and the accuracy of the mobile application notifica- 
tion functionally was 96.88%. 


Challenges and Future Trends 


Technology is transforming healthcare as it brings new promises. Individuals rely 
on Quantified Self (QS) [92] technologies to collect multiple types data, such as 
sleep, location, mobility, and physical activity (including medication taking activ- 
ity). However, still there are some technological challenges that need to be addressed 
in order for these systems to make a broader impact. As highlighted in Table 3.2, 
some weakening factors that may limit the adoption of such systems are the accu- 
racy, energy consumption, and acceptability. However, there are other factors that 
are respectively related either directly or indirectly to these main factors such as 
lifetime, data fidelity, and user’s comfort. Discussed below are these challenges and 
highlights on the trade-offs between them. 


Challenges 
System Accuracy and Data Fidelity 


Achieving better healthcare requires accurate systems that capture the user’s activity. 
This also applies to adherence monitoring systems. In general, accuracy is deter- 
mined by the device being used for capturing the medication taking activity. 
Furthermore, the setting of medication taking can affect and limit the technology 
advances in use. For example, the system might operate at low-sampling rates as a 
trade-off for energy consumption minimization. However, this comes at the cost of 
lower data quality. Accuracy includes data quality, data precision, or data fidelity [67]. 

Data fidelity can be characterized by the sampling frequency, the sensor opera- 
tion mode, and the duty cycling. Obtaining high accuracy data demands the system 
to be running at high-fidelity. However, high-fidelity systems deplete the battery 
energy at a fast rate, as their core should be set to run frequently for capturing the 
monitored event precisely. Thus, when engineering a tracking system, the energy 
consumption management should be considered carefully. 
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Energy Consumption and Lifetime 


Monitoring systems can be battery-powered, for example, in the case of sensor net- 
works and mobile device-based systems. This poses a challenge as the battery has 
limited energy budget [93]. From a system point of view, it is anticipated that a suf- 
ficient amount of electric current is being fed to the system to ensure its functional- 
ity. At the same time, from a user point of view, it is expected that the system 
lifetime lasts for as long as possible as application developers must either frequently 
replace batteries or use rechargeable batteries. This would likely be inadequate for 
user’s acceptance and costly [94]. 

Even though only rare studies focused on the energy consumption of medication 
adherence monitoring systems, this is still central in this context as it can severely 
affect the performance and efficiency of the system [95]. This can be imagined by 
taking wearable systems powered by non-rechargeable batteries as an example. In 
general, the battery is a complex system that can behave unpredictably when 
affected by several factors and conditions, including the temperature and the applied 
load [96]. High-fidelity motion sensors are utilized within wearable devices for 
accurately sensing and quantifying the motion associated with medication taking 
activity. However, there is a trade-off between energy consumption and data fidelity. 
On the one hand, the sensor device should be operating continuously and sampling 
data frequently. On the other hand, even if temperature conditions are perfect, 
enabling the sensor(s) for frequent data sampling results in increasing the internal 
resistance of the battery and affecting its chemical and physical properties [97]. 
Operating the battery under such timing and intensity conditions will not enable it 
to provide voltage at a sufficient level that operates the connected device correctly, 
even with a considerable amount of unused charge being left. As a consequence of 
the experienced discharge behavior, the system’s lifetime is directly affected. As 
such, wise battery usage is required [98]. Thus, techniques such as collaborative 
sensing to be employed for minimizing energy depletion in such systems. Once the 
energy consumption issue achieves notable progress, battery-powered systems such 
as wearable and portable systems can be used more widely in the area of adherence 
monitoring applications. 


Acceptability and User’s Comfort 


The user’s perception of a monitoring system has a great impact on its adoption and 
success. First, technological barriers such as battery energy consumption, mobility 
support, and others play a significant role as barriers to the wide acceptance of 
technology-based systems. Second, ethical challenges such as privacy and confi- 
dentiality also exist. Users are concerned about behaviors being monitored beyond 
medication taking and the potential of unintended users accessing the information 
collected [99]. In addition, users, and especially the older ones, tend to have social, 
physical, demographic, and cultural barriers towards using technology and, as a 
result, barring the user’s acceptance of modern technology [41, 74]. 
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Tampering, Authentication, and Active Non-Compliance 


Two key challenges arise because users may try to actively deceive the system into 
thinking they are compliant when they are not. Tampering occurs when an unau- 
thorized user receives the medication. The first challenge then becomes one of 
authentication—Is the person who is taking the medication who he claims to be? 
Tampering can arise for medications which can become addictive, such as opiates, 
where an addict or dealer has an incentive to fool the system. Authentication and 
authorization are analogous concepts in computer security—Is the person who they 
claim to be, and is this person authorized to take the medication? Although few 
projects have specifically tackled these security challenges, an array of wearables 
has investigated if a wearable is actually worn by the person it is supposed to [100]. 
A second set of approaches attempts to prevent unauthorized access with the use of 
physical barriers, such as locks on the pillboxes. A related set of approaches does 
not try to prevent unauthorized access, but rather take an auditing approach. For 
example, learning the wrist motions of different people can create an audit trail 
[101], which can then be used to identify tampering for later remediation. 

The second challenge is observing active non-compliance, which is when a legit- 
imate user actively deceives the system. Such behavior can occur when a user dis- 
agrees with a medical professional's treatment but appears to comply rather than 
challenge the professional's judgment. Active deception on the part of the user is 
more difficult to solve as the person using the system is legitimate but chooses not 
to consume the medication. A variety of approaches can be employed, such as video 
monitoring, but simple actions, such as placing medication in the mouth, faking a 
swallow, and then spitting it out later, will deceive most current technologies. The 
recent proposal of Quality of Life technologies [102] can help in monitoring differ- 
ent aspects of the individual's life. These can include social relationships and envi- 
ronment monitoring, that may impact the psychological and social factors and in 
result, patient's behavior. Creating monitoring systems that correctly identify active 
non-compliance remains an important research challenge. 


Future Trends 


Itis clear from this review that most solutions have some sort of limitation. As such, 
the developed system may harness the advancements of a combination of technolo- 
gies to achieve the ultimate goal. However, overcoming the challenges that were 
previously mentioned can be achieved as follows. To precisely monitor patient 
adherence, fine-grained sensors such as load cells, motion sensors for detecting and 
classifying gestures associated with hand-to-mouth movement, and sensors for cap 
opening and closure verification, are strong candidate technologies. 

The integration of sensors that consume very little energy with limited fidelity 
along with sensors that report much higher fidelity of activity but also power-hungry 
on a single platform and decide what sensor and when to have it on, is an example 
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of collaborative sensing that can be harnessed for prolonging the lifetime of a 
battery-powered system [67]. However, this requires sensor fusion algorithms that 
build a unified model based on different sensed and reported inputs—for example, 
Bayesian inference. In addition, since the wireless functionally in wireless-enabled 
systems constructs a bottleneck as it consumes a large portion from the battery 
energy, searching for low communication technologies is a must. An example of 
this can be the Transmit Only (TO) approach [67, 103] that can be employed rather 
than WiFi or Bluetooth. The TO technique is a single hop communication that does 
not demand handshaking or acknowledgment, and thus it minimizes the energy con- 
sumed for packet transmission to only a few tens of micro joules [67]. Finally, user’s 
acceptability and comfort might be achieved by carefully designing a pill container 
that is low-energy consuming, smart, and wireless. 


Conclusions 


Medication non-adherence is a major problem in the healthcare sector. Poor medi- 
cation adherence leads to healthcare resource wastage and sub-optimal treatment 
outcomes. As such, it has become an attractive research area for many researchers 
from multidisciplinary domains with the aim of developing new monitoring and 
interventions that can detect and correct medication taking regimens once they devi- 
ate. In this chapter, we have covered the technology-based techniques and systems 
for medication adherence monitoring. In addition, we put special stress on the ad- 
vantages, disadvantages, and challenges associated with these approaches, but how 
those translate into changed operational and clinical outcomes requires more feed- 
back and observations of both patients and clinical practitioners. From this review, 
we can conclude that work is still required to enhance technology-based systems 
that can overcome these challenges, especially the accuracy, user comfort, and bat- 
tery consumption. In addition, assuring the whole workflow with minimal burden 
for the patients and health practitioners is still to be met. 
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Introduction 


There are many ways to conceptualize “Energy” and “Fatigue” in the context of the 
WHO Quality of Life domain [1]. Energy and fatigue may be interrelated but may 
also be considered orthogonal. Low energy can be characterized by fatigue, lack of 
motivation, and lack of interest, while states of excessive energy can reach patho- 
logical levels that include disrupted sleep, restlessness and agitation, or even mania 
[2]. Although lacking energy can be burdensome and uncomfortable, it is simulta- 
neously an adaptive symptom that is perceived as a need to rest or slow down [3]. 
Given that energy is a valuable resource, efficient spending and conservation of 
energy may result in the greatest chances of vitality and even survival [4]. Curiously, 
one tends to think of energy as a resource that is depleted and then replenished with 
rest. It is also frequently observed that using energy may be synonymous with gen- 
erating energy (e.g. one may feel more replenished after engaging in an activity than 
they do following rest). This curious paradox highlights the potential value of clas- 
sifying and enhancing our understanding of energy. Fatigue is both a normative 
experience as well as associated with many chronic illnesses and psychiatric disor- 
ders. Fatigue can be characterized by subjective feelings of “tiredness” and “lack of 
energy" [5] and can serve as a signal to prevent strain, damage, and injury [6]. 

In this chapter, energy refers to the strength and vitality required for sustained 
physical or mental activity. Lack of energy or fatigue is used to describe the subjec- 
tive sensation (perceived fatigue) as well as the objective and quantifiable change in 
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performance (fatigability) [7]. Fatigue can be classified as pathological or non- 
pathological. Pathological fatigue can be described as an overwhelming sense of 
tiredness at rest, exhaustion with activity, lack of energy that precludes daily tasks, 
or loss of vigour [7]. In healthy adults, non-pathological fatigue is predictable and 
does not interfere with usual daily activities. Non-pathological fatigue is typically 
brought about by prolonged exertion and diminishes with rest [8]. In addition to 
pathological and non-pathological fatigue, fatigue may also be subdivided as either 
physical or mental (cognitive/psychiatric) and further subdivided as primary (neu- 
rological) or secondary (non-neurological) [7—10]. Furthermore, performance refers 
to an individual’s functioning in their daily environment while capacity refers to the 
maximal or optimized level of functioning. 

Preliminary studies were conducted on energy and fatigue during the First World 
War when researchers investigated the impact of fatigue on efficiency and produc- 
tivity of the industrial workforce [11]. This “occupational fatigue” continues to be a 
focus of research attention, especially in vocations and occupations in which fatigue 
carries serious implications. Traditionally, energy and fatigue have been assessed 
using qualitative, self-reported outcomes [12] and can be obtained from a number of 
validated scales [13, 14]. Most clinical fatigue studies use self-report measures that 
can broadly be classified as measuring perceptions of fatigue [8]. Despite the numer- 
ous scales that measure fatigue, there is no agreed-upon standard of which to com- 
pare subjective reports of fatigue [15, 16]. 

The use of technology to monitor and manage energy and fatigue has been inves- 
tigated in order to help healthy individuals continue to live healthily [3, 6, 17], assist 
individuals with health issues [18—20], and address vocational or occupational 
fatigue to improve personal and workplace safety [21-23]. The monitoring of 
energy and fatigue helps individuals adapt their effort in recreational (e.g., amateur 
sport, exercise) and occupational (e.g., drivers, pilots, police officers, professional 
athletes, shift workers) settings to prevent negative effects (e.g., burnout, exhaus- 
tion, accidents, injury) and maintain quality of life [24, 25]. Further, energy and 
fatigue research is needed to examine their connection to underlying or potential 
health conditions as well as interventional studies to validate the operationalization 
of energy and fatigue monitoring in daily life. 

In this chapter, we will classify energy and fatigue and present their measure- 
ment. The chapter is structured as methods of our work, classification of energy and 
fatigue (pathological as well as non-pathological), measurement and assessment of 
energy and fatigue, discussion of results, and conclusive remarks. 


Methods 


We conducted a scoping review of the existing literature between 2010 and 2020 in 
Google Scholar on the technology-enabled assessment of energy and fatigue. Search 
terms related to energy and fatigue (e.g., "fatigability", “tiredness”) were coupled 
with terms pertaining to each of the following domains: (1) the population under 


study (e.g., "athlete", “driver’), (2) the health outcomes (e.g., "circulation", 
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Table 4.1 Domains of energy and fatigue literature review 
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Domain Inclusion | Rationale Search terms (selective) 

Energy / Mandatory | Energy and fatigue are often | Energy, fatigue, fatigability, 

fatigue proxied by synonyms or tiredness, vitality 
antonyms. 

Population Optional | Papers assessing energy and | Athlete, driver, performance, pilot, 
fatigue in healthy individuals | police, shift, sport, worker, 
often focus on a specific employee 
segment of the general 
population. For instance, two 
areas of focus are athletics 
and occupational fatigue. 

Health Optional | Health outcomes are often Cancer, cardiovascular, circulation, 

outcomes delineated by specific dementia, heart, kidney, mental, 
elements of human physiology | pulmonary, respiration 
or pathology: Organs, 

Systems, processes, and 
diseases. In addition, such 
elements can be further 
delineated by the population 
segment under study. 

Measurement | Optional | Methodological measurements | Accelerometer, app, application, 
using technology can be camera, band, ecological 
described by the procedure, momentary assessment, 
device, sensor, process, or performance, capacity, 
result. electrocardiogram, 

electrooculogram, experience 
sampling method, Fitbit, galvanic, 
mobile, sensor, smart band, 
smartphone, smartwatch, vision, 
watch, wearable 


^^ 66 


"dementia", “heart’’), and (3) the measurement (e.g., “accelerometer”, "electrocar- 
diogram", *wearable"). One example search phrase was “galvanic energy fatigue 
tiredness vitality”. We also reviewed the relevant references of the identified litera- 
ture. Table 4.1 reviews the domains, search terms (selective), and the rationale for 


choosing the domain. 


Results 


We found 40 reviews on energy and fatigue pertaining to the domains and 60 studies 
assessing fatigue by using technology. The search results included in this review 
either (1) reviewed energy and fatigue assessment for a specific population and/or 
health outcome, (2) provided evidence for the use of measurement to monitor or 
manage energy or fatigue, or (3) discussed human factors of technology towards 
monitoring energy and fatigue. The taxonomy of fatigue resulting from our litera- 
ture review is depicted in Fig. 4.1. 
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Pathological [ Non-Pathological | 
Physical Mental Physical Mental 
Neurological [Non-neurological 
Central Cognitive Central Cognitive 


Peripheral Affective Peripheral Affective 


Fig. 4.1 Taxonomy of fatigue with pathological and non-pathological types [7—10, 26] 


Energy and Fatigue Classification 


Pathological Fatigue 


Pathological fatigue is prolonged or chronic (26 months), can be highly debilitating, 
and is much less common than non-pathological fatigue [27]. Pathological fatigue 
may be best understood as an amplified sense of normal (non-pathological) fatigue 
that can be induced by changes in one or more variables regulating work output [9]. 
For instance, a healthy individual may experience fatigue during or after exercising, 
but the same individual may perceive even more fatigue when exercising during an 
infectious disease [7]. Diseased individuals describe fatigue as an overwhelming 
sense of tiredness at rest, exhaustion with activity, loss of vigour, or lack of energy 
that precludes daily tasks, inertia or lack of endurance [28]. Pathological fatigue 
may be classified as physical or mental and is associated with multiple illnesses. 


Physical Fatigue 


Pathological physical fatigue includes neurological and non-neurological fatigue. 


Neurological Fatigue 


Neurological fatigue suggests that the physical expression of fatigue is mediated by 
central and peripheral mechanisms [27]. Therefore, neurological fatigue may be 
further classified as central or peripheral [9]. 

Central fatigue is generated at sites proximal to the peripheral nerves and referred 
to as a progressive decline in the ability to activate muscles voluntarily [29]. Central 
fatigue is due to impaired muscle performance that arises from the central nervous 
system [28]. A feeling of constant exhaustion is a characteristic of central fatigue 
[9]. Pathological central fatigue is found in Multiple Sclerosis, Traumatic Brain 
Injury, Parkinson's Disease, and many others. 
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Mechanisms of peripheral fatigue are usually attributable to a neuronal or mus- 
cular origin. Peripheral fatigue results from a lack of response in the neuromuscular 
system after central stimulation [27]. Peripheral fatigue is characterized by the fail- 
ure to sustain the force of muscle contraction [9]. Pathological peripheral fatigue is 
found in neuromuscular disorder, rhabdomyolysis, muscle ischemia, restless legs 
syndrome and more. 

In many of the previously mentioned health conditions, physical inactivity is a 
contributing factor to the increased fatigue of the patient [30]. Deconditioning, as a 
result of restricted physical activity, results in large decreases in muscle mass and 
strength, as well as increased fatigue due to changes in muscle metabolism [31, 32]. 
Physical fatigue is also increasingly observed as a secondary outcome in many dis- 
eases and health conditions during the performance of everyday activities [32]. 


Non-Neurological Fatigue 


The exact mechanism of how non-neurological disease causes fatigue is not fully 
understood [7]. However, there are indications that peripheral proinflammatory 
cytokines signal the central nervous system to initiate fatigue [33]. A common non- 
neurological cause of temporary fatigue is an infection or the common cold. Non- 
neurological causes of chronic fatigue include infectious diseases (human 
immunodeficiency virus, mononucleosis, Borreliosis, and chronic pancreatitis), 
hematologic disease (anaemia and hemochromatosis), dehydration, immunological 
disease (celiac disease), rheumatological disease, cardiac disease (heart failure and 
cardiomyopathy), endocrinologic disorder (diabetes, Addison’s disease, hypopitu- 
itarism, and hypothyroidism), renal disease (insufficiency and dialysis), lung dis- 
ease (chronic obstructive lung disease and asthma), malnutrition (poor diet, irritable 
bowel disease, eating disorders and hypoproteinemia), liver disease, chronic pain, 
chronic fatigue syndrome, fibromyalgia, malignancy (cancer, sarcoma, lymphoma, 
and leukaemia), Gulf War disease, poisoning, mineral or vitamin deficiencies, 
drugs, or irradiation [7]. 

Drugs and medications may also be a cause of non-neurological fatigue. The 
drugs that cause fatigue include alcohol, antihistamines, benzodiazepines, antispas- 
modics, antiepileptic drugs, neuroleptics, and narcotics [7]. 


Mental Fatigue 


Mental fatigue in the pathological domain includes cognitive and affective (psycho- 
logical/psychiatric) fatigue. Cognitive fatigue has been studied in the context of MS 
[34], cancer [35], TBI [36], HIV [37], and other diseases. Affective fatigue is influ- 
enced by psychological factors (attitude, motivation, will, endurance, flexibility, 
inertia, persistence, concentration, and alertness) as well as psychiatric factors 
(depression, mania, psychosis, and addiction) [28]. Individuals with chronic fatigue 
report poorer mental health than their non-chronic fatigue counterparts [38]. 
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Non-pathological Fatigue 


In contrast to pathological fatigue, non-pathological fatigue is short term and remits 
with rest. Non-pathological fatigue is sometimes referred to as physiological fatigue 
in the scientific literature. Non-pathological fatigue alerts the individual to opportu- 
nity costs of current activities, as well as of the attraction of alternative activities 
[39]. Fatigue in healthy individuals is a universal experience and a natural occur- 
rence after physical or mental efforts, usually relieved by rest. Research has exam- 
ined biological explanations for pathological versus non-pathological fatigue [40], 
as well as self-report scales to distinguish fatigue associated disease from fatigue 
associated with healthy controls [41]. It has been reported that 55% of healthy indi- 
viduals identified a physical sensation of fatigue and 24% identified a mental sensa- 
tion of fatigue [26]. 


Physical Fatigue 


From a physical perspective, fatigue is described as the inability of the muscles to 
maintain the required level of strength during exercise activities [42, 43]. It can also 
be characterized as an exercise-induced reduction in muscle’s capability to generate 
force. There is no single cause of physical fatigue [44] and physical fatigue includes 
both central and peripheral fatigue. 

Central fatigue designates a decrease in voluntary activation of the muscle, 
whereas, peripheral fatigue indicates a decrease in the contractile strength of the 
muscle fibres and changes in the mechanisms underlying the transmission of muscle 
action potentials [45]. Central and peripheral fatigue is a common experience dur- 
ing sport and exercise activities. 

The impact of physical fatigue on cognitive performance depends both on the 
intensity and the duration of the exercise [46, 47]. Prolonged physical exercise lead- 
ing to dehydration and physical fatigue is associated with a reduction in cognitive 
performance [48]. 


Mental Fatigue 


Mental fatigue includes cognitive and affective fatigue and is an unfocused men- 
tal state, characterized by distraction, frustration, or discomfort. Mental fatigue is 
a psychobiological state caused by prolonged periods of demanding cognitive 
activity and characterized by subjective feelings of “tiredness” and “lack 
energy” [4]. 

In terms of cognitive activities, mental fatigue may be defined as the percep- 
tion of feeling cognitively fatigued after performing demanding cognitive activ- 
ities that involve concentration, attention, endurance, or alertness [49]. In the 
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cognitive domain, fatigability can be measured as a decline in the reaction time, 
a decline in accuracy on continuous performance tasks, or a probe task that is 
given before and immediately after a fatiguing cognitive task [50, 51]. This 
cognitive fatigue is associated with problems completing tests, particularly 
where there is a requirement to sustain high levels of effort over time [39]. The 
effects of mental fatigue on cognitive performance [4, 51—53], and the skilled 
performance of drivers [54] and air pilots [55], have been investigated. Mental 
fatigue also limits physical performance [56] through perceived exertion [5]. 
Similarly, mental fatigue, following the performance of cognitive tasks, impairs 
emotion regulation [57]. 

Affective fatigue is characterized by low mood, tiredness, weariness, and leth- 
argy [39]. It has been reported that 21% of healthy individuals identified an affective 
sensation of fatigue [26]. Non-pathological affective fatigue includes self-regulatory 
fatigue, empathy fatigue, and other fatigue associated with emotional depletion 
(burnout). 


Factors Influencing Fatigue 


Pathological and non-pathological fatigue is influenced by numerous factors, such 
as age, gender, physical condition, diet, latency to last meal, mental status, psycho- 
logical conditions, personality type, life experience, and the health status of the 
individual [7]. Most studies found more fatigue in women than in men [38, 58-62]. 
Inconsistent findings have been reported regarding age and fatigue [38, 58, 62, 63]. 
Additionally, a high level of formal education has been associated with a lower 
prevalence of fatigue [61, 64, 65]. 

Sleepiness and fatigue are distinct and interrelated. Sleepiness refers to an 
increased propensity to fall asleep [66], while fatigue refers to tiredness resulting 
from exertion or illness. Fatigue may be regarded as a motivational drive to rest [67] 
and non-pathological fatigue will usually remit with rest. Sleepiness is related to 
circadian and homeostatic influences and remits after sleep [68], but not after rest. 


Energy and Fatigue Measures 


Fatigue perception is frequently measured by self-report scales, while fatigability is 
frequently assessed by performance, capacity, and technology-reported measures 
[69]. Subjective measures include scales and prompts for assessment while objec- 
tive measures include performance and capacity tasks (physical and cognitive), 
physiological measurements (cardiac, ocular, neural), and markers (biological and 
behavioural). 


86 N. L. Solomon and V. Manea 
Subjective Measures 


Fatigue perception is frequently measured by application of patient-reported outcomes 
(PRO) [12] through validated scales prompted for assessment. These scales may be 
administered momentarily, daily, monthly etc. through paper, web, or smartphone. 


Scale Instruments 


Scales for self-reporting may be unidimensional, evaluating a single property, or 
multidimensional, evaluating multiple properties [49]. These instruments address 
different aspects of fatigue and energy and some address more than one aspect. No 
single measure of fatigue adequately captures the complexity of the phenomenon 
[15]. Researchers have pointed out that “in developing fatigue scales, there is a 
“catch 22” situation: before a concept can be measured, it must be defined, and 
before a definition can be agreed upon, there must exist an instrument for assessing 
phenomenology. There is, unfortunately, no “gold standard” for fatigue, nor is there 
ever likely to be” [13]. Table 4.2 in this section depicts several scale instruments 
routinely used to measure energy and fatigue. The majority of these energy/fatigue 
self-report scales were designed for pathologic populations, but have been applied 
to non-pathologic populations as well. 

Considerations in choosing a particular scale include recall period, unidimen- 
sionality or multidimensionality, scale structure and length, and suitable population. 
Scales differ in their scope, some measuring severity only, and others duration and 
impact on a range of functions [14]. Fatigue measures have been evaluated for the 
number of symptoms assessed, dimensions of fatigue explored, the time frame of 
the assessment, scale, method, the population on which the scale was developed, 
and psychometric properties [13, 14]. 

Some applications of these scales are illustrated below. SF-36 and PROMIS have 
been used in traditional studies assessing fatigue in the general population [14, 81]. 
FQ, FSS, and MAF have been employed to assess workplace-related fatigue [82, 
83]. POMS has been used to assess fatigue in bus drivers [84] and sport athletes 
[24]. Scales were also used in traditional studies to assess energy and fatigue in 
individuals with a plethora of diseases, e.g., cancer [85, 86], cardiovascular disease 
[87, 88], chronic obstructive pulmonary disease [89], diabetes [90], fibromyalgia 
[91], hearing loss [16], inflammatory bowel disease [92-94], lupus [95], major 
depressive disorder [96], multiple sclerosis [97—99], psoriasis [100], pulmonary 
arterial hypertension [101], renal disease [102], rheumatic disease [103, 104], sleep 
apnea [105], stroke [106, 107], and traumatic brain injury [10]. 

Smartphone collection of self-reported energy and fatigue data has been utilized 
in the context of multiple sclerosis [108], cancer-related fatigue [109], and bipolar 
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Table 4.2 Scale instruments routinely used to measure energy and fatigue 
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Recall 
Instrument period | Measures Administration | Usage 
Fatigue assessment | Usually | Unidimensional; fatigue | 10 items, 5-level | Pathologic and 
scale (FAS) [70] ("refer severity Likert scale non-pathological 
to how (developed for 
you chronic fatigue) 
usually 
feel") 
Functional Past Unidimensional; 13 items, 5-level | Pathologic 
Assessment of week general fatigue Likert scale (people with 
Chronic Illness various chronic 
Therapy (FACIT-F) illnesses, 
Fatigue Subscale including cancer) 
[71] 
Fatigue impact scale | Past Multidimensional; 40 items, 5-level | Pathologic 
(FIS) [72] month, | physical, cognitive, and | Likert scale (developed for 
present | psychosocial infectious disease 
time functioning, total patients) 
fatigue 
Fatigue Past Multidimensional; 11 items, 4-level | Pathologic and 
questionnaire / month  |physical, mental, total, | Likert scale non-pathological 
fatigue scale (FQ / substantial, transient, (developed for 
FS) [73] and chronic fatigue use in hospital 
and community 
populations) 
The fatigue severity | Past Unidimensional; fatigue | 9 items, 7-level | Pathologic and 
scale (FSS) [74] week severity Likert scale non-pathological 
(developed for 
patients with 
multiple sclerosis 
or systemic lupus 
erythematosus) 
Multidimensional Past Multidimensional; 16 items, Pathologic and 
assessment of week degree, severity, 4—10-level non-pathological 
fatigue (MAF) [75] distress, and impact of | Likert scales (developed for 
fatigue patients with 
rheumatoid 
arthritis) 
The Lately | Multidimensional; 20 items, 7-level | Pathologic and 
multidimensional ("refer | physical, mental, and Likert scale non-pathological 
fatigue inventory tohow | general fatigue; reduced (used in 
(MFI) [76] you have | activity and motivation chronically 
been unwell and well 
feeling populations) 
lately") 


(continued) 
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Recall 
Instrument period | Measures Administration | Usage 
Medical outcomes Past Multidimensional; 4 items, Pathologic and 
study short form month  |physical, cognitive, 3-6-level Likert | non-pathological 
(SF-36) energy and social, and emotional scales and yes/ | (developed to 
fatigue subscale [77] functioning no measure the 
health status of 
individuals living 
in the 
community) 
Patient-reported Past Multidimensional; Up to 95 items, | Pathologic and 
outcomes week physical, mental, 5-level Likert non-pathological 
measurement general, emotional, scale (can reliably 
information system total, substantial, estimate fatigue 
(PROMIS), fatigue transient, chronic reported by the 
short form or fatigue; reduced activity U.S. general 
computerized and motivation; population) 
adaptive test [78] physical, cognitive, 
psychosocial, social, 
emotional functioning; 
energy 
Profile of mood Past Multidimensional; 65 items, 5-level | Non-pathological 
states (POMS), week, physical and mental Likert scale (adult version and 
fatigue and vigour | present | fatigue; energy adolescent 
subscales [79] time version) 
Visual analog scale | Present | Bidimensional; energy | 18 items, visual | Pathologic and 
to evaluate fatigue time: and fatigue analogue non-pathological 
severity (VAS-F) “Right (validated with 
[80] now” adults aged 
18-55 years) 


disorder [110]. Smartphone data collection often incorporates validated scales. For 
example, a mobile phone application to collect data on self-reported fatigue for 
multiple sclerosis [108] incorporated PROMIS. Researchers concluded that a phone 
application incorporating PROMIS may be useful to provide estimates of fatigue to 
facilitate clinical monitoring of fatigue for clinic settings. 


Momentary Assessments 


The Ecological Momentary Assessment (EMA) is a technique that elicits a repeated, 
real-time measurement of behaviours or experiences as they occur in the naturalistic 
setting of an individual’s daily life. This method was originally developed to per- 
form in situ data collection for behavioural medicine [111]. The Experience 
Sampling Method (ESM) aims to assess participant thoughts, behaviours, and feel- 
ings during daily life by collecting self-reports, triggered at various moments during 
the day [112]. The two terms (EMA and ESM) are used interchangeably, and in 
practice, they are measured using the same methods [113]. 
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Traditional studies employing EMA/ESM assessed fatigue and fatigability in 
segments of the general population. For instance, this method has been applied to 
demographic groups, work settings, and disease populations. Specifically, the rela- 
tionship between women’s passion for physical activity and vitality was examined 
using SF-36 scale [114]. Researchers have also employed POMS scale to examine 
occupational energy management strategies by hourly diary questions in academic 
workers [115]. A separate study examined the effects of breaks on regaining vitality 
in the workplace using an activation-deactivation adjective checklist [116]. 
Additionally, EMA/ESM assessment of energy/fatigue has been applied to disease 
populations including osteoarthritis ((117] researchers used SF-36 scale), kidney 
disease ([45] researchers used Daytime Insomnia Symptom Scale), and cancer 
([118] researchers used a single-item fatigue intensity scale; [20] researchers used 
10-point Likert scale for current fatigue). 

Mobile-administered EMA/ESM has been applied to the management of dis- 
eases. For cancer and its treatment, fatigue is one of the most common and distress- 
ing side effects. Cancer-related fatigue causes disruption in all aspects of Quality of 
Life and may be a risk factor for reduced survival [119]. A mobile phone-based, 
symptom management system can assist in the management of chemotherapy- 
related toxicity in patients with breast, lung and colorectal cancer [109]. This sys- 
tem prompts patients to complete an electronic symptom questionnaire on their 
mobile phone twice a day. A systematic review of mobile apps for bipolar disorder 
[110] identified thirty-five symptom monitoring apps aiming at assisting users with 
symptom tracking. 


Objective Fatigue and Energy Measures 


Fatigability is primarily measured by quantifying the decline in one or more aspects 
of performance during the continuous performance of a prolonged task or compar- 
ing performance before and immediately after a prolonged performance of a sepa- 
rate fatigue-inducing task [8]. In pathological cases, individuals may experience 
fatigue even in activities of daily living [120]. When objectively measuring fatigue, 
it is important to indicate the domain examined and the task used to induce 
fatigability. 

Fatigue-related decrements in task performance can be measured by following 
two common approaches. Ackerman [121] provides a classification of procedures 
for cognitive fatigue, and we argue that these same approaches pertain to physical 
fatigue as well. The indirect approach consists of the assessment of cognitive ability 
before and after a prolonged period of time during which effort may vary. The direct 
approach consists of the continuous measurement of fatigue during the difficult 
task. The benefits of the first method are that all participants can complete the same 
task, while the variation lies in the difference between ex-ante and ex-post fatigue 
among individuals. This method does not quantify the performance decrease as a 
continuous function of time. Conversely, the second method can monitor fatigue 
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accumulation, but the tasks may vary. One example is vigilance tasks, where partici- 
pants are required to maintain attention for target events while ignoring other stim- 
uli (122, 123]. 

There is a distinction between capacity (describing a person’s ability to execute 
a task in a standardized, optimized, or controlled environment), capability (describ- 
ing what a person can do in their daily environment), and performance (describing 
what a person actually does in their daily environment) [124]. Capacity is the com- 
posite of all the physical and mental capacities that an individual can draw on and 
performance is what individuals do in their current environment [125]. It is beyond 
the scope of this chapter to classify past studies as capturing capacity, capability, 
performance, or a combination. 


Physical Assessment 


The monitoring of fatigue and energy has been examined as an approach to maintain 
health, assist in disease management, and improve performance, productivity, and 
safety. A plethora of methods have been employed in order to monitor fatigue and 
energy: performance-reported outcomes (PerfRO) [12] for physical and cognitive 
fatigability, and tech-reported outcomes (TechRO) [12] from physiological pro- 
cesses (cardiac, ocular, neural) and markers (biologic, behavioural). 

Fatigability is usually quantified as a decline in peak force (torque), power 
(velocity of muscle contraction), speed, fatigue index (force change over time), 
sense of effort, perception of effort, or accuracy of performance after performing a 
task, which requires physical effort [7]. Characteristics of tasks include exercise 
type, intensity, load, tested muscle, and physical environment [28]. 

The first dimension of physical performance fatigue is “physical capacity” (1.e., 
maximum performance). The two most common indicators of physical capacity are 
(1) the aerobic capacity and (2) the power output capacity. Measures of aerobic 
capacity include the maximal oxygen volume (VO;-max). Measures of power out- 
put include the peak power output. Momentary exercises leading to the assessment 
of these measures include aerobic and resistance training [98]. Example exercises 
routinely used, e.g., in professional sports players include various jump protocols, 
including squat and countermovement jumps, which can lead to indirect assess- 
ments of fatigability [24]. Direct measures of fatigue include a joint range of motion 
or flexibility of appendages such as the knee, hip, groin, and other joints during the 
exercise. 

The second measured dimension of fatigue is *muscular strength." Studies mea- 
suring muscular strength included momentary resistance training of various types 
(weight machines, free weights, resistance bands, cycling ergometers) and other 
strength training (specialized locomotor training, cycling, aquatics) [98], muscular 
oxygen consumption (mVO,), or electromyography (EMG). 
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The third dimension of fatigue is “mobility,” which is more commonly measured 
in cases of pathological fatigue. Mobility measures include the momentary 6-Minute 
Timed Walk (6MTW) [126], the Timed 25-Foot Walk [127], and the Timed Up & 
Go [128]. 

Exercise-specific hardware used for such exercises include treadmills, weight 
machines, free weights, and resistance bands. Technology-enhanced exercises 
include the robotic-assisted treadmill and functional electrical stimulation-assisted 
cycling [98], and transcranial magnetic stimulation [7]. Figure 4.2 depicts an exam- 
ple of hardware used to measure physical performance. 

Studies assessing non-pathological physical performance as a proxy for physical 
fatigue involved segments of the general population, e.g., physical fatigue in young 
adults using POMS, trail-making test on an iPad and mVO, [129], physical fatigue 
during a sit-to-stand physical test by using EMG and accelerometer (Samsung) in 
the lab [130], or PhysioLab, a physiological computing toolbox measuring multiple 
signals (ECG, EMG, and EDA) to study cardiorespiratory fitness in elderly popula- 
tions [131], all momentary. 

Other observational studies assessed physical fatigue in a pathological context 
with individuals with health conditions or diseases; assessments include the effects 
of caloric restriction on cardiorespiratory fitness and fatigue in older adults with 
obesity by using graded exercise tests measuring VO,-max [132], the differences in 
motor fatigue between patients with stroke and patients with multiple sclerosis by 
using self-reported SF-36 and 6MWT [133], physical fatigue in lumbar disc hernia- 
tion by using EMG [134]. 

Continuous monitoring studies assessed the effects of disease on fatigue, e.g., a 
rehabilitation program on aerobic fitness, cancer-related fatigue, and quality of life 
using subjective MFI and objective energy expenditure armbands (SenseWear) 
[135], or the fatigue monitoring system (FAMOS) which can monitor physiological 
parameters from multiple sclerosis patients and controls, all pathological. 


Fig. 4.2 Hardware for 
physical performance 
example (ergometer). © 
Laufband Ergometer by 
Robowalk licensed under 
CC-BY-SA-4.0 
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Cognitive Assessment 


In the cognitive domain, fatigue leads to the degradation of cognitive performance 
[122], as reflected by degradations in verbal, visual, short, and long-term memory, 
processing speed, primary and divided attention, verbal fluency, motor speed, read- 
ing speed, visual scanning, orientation, calculation, success rate, and other measures. 

Cognitive assessments were measured by using numerous momentary measures, 
which collectively assess the above degradations. Table 4.3 reviews several task- 
based tests yielding cognitive performance-reported outcomes [12]. 

Cognitive performance studies included fatigue assessment in non-pathological 
segments of the general population, e.g., alertness, vitality, and sleepiness by using 
Psychomotor Vigilance Task (PVT) and other tasks in different lighting settings 
[146], occupational fatigue, e.g., in healthcare and medical staff by using the rate of 
error [147], or airline pilots on the flight deck by using PVT [148]. 

Cognitive performance studies also included fatigue assessment in pathological 
settings, e.g., the relationships between health-related Quality of Life, fatigue, and 
exercise capacity in coronary artery disease individuals using MFI and a bicycle 
ergometer test [149]. 

Technology-driven studies include assessments of mental fatigue in a non- 
pathological context by performing tasks with a computer, e.g., keyboard and mouse 
interaction patterns [150] recovery from work exhaustion by use of Twitter [151], or 
in a pathological context. For example, those living with an acquired brain injury 
often have issues with cognitive fatigue due to factors resulting from the injury. 
Studies have shown fatigue to be one of the most disabling symptoms, regardless of 
the severity of brain injury [152—154]. Researchers presented a smartphone applica- 
tion for the evaluation of cognitive fatigue, which can be used daily to track cogni- 
tive performance in order to assess the influence of fatigue [155]. Researchers 
concluded that the presented smartphone application for the evaluation of cognitive 
fatigue could be utilized in everyday life. 


Cardiac Physiology 


Cardiac activity measures used to assess fatigue include the resting heart rate (HR), 
exercise heart rate (HRex), heart rate variability (HRV), and the heart rate recovery 
(HRR). The heart rate may increase or decrease in response to a variety of factors 
including physical and mental effort, distress, and anxiety that are potentially asso- 
ciated with fatigue [16]. Elevated HRV was observed during strenuous tasks in indi- 
viduals with chronic fatigue [156] and healthy individuals of young age while 
performing a task [157]. HRR may serve as a marker of acute training-load altera- 
tion, however recent studies showed inconclusive results [24]. A more detailed mea- 
sure of heart activity is the electrocardiogram (ECG), an electrophysiological 
method, which records the electric signals of the heart and from which the HR can 
be derived. Figure 4.3 depicts an electrocardiogram with an electrocardiograph and 
electrodes placed on the human body. 
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Table 4.3 Tasks and measures of cognitive performance 
Task Measures Administration Usage 
Mini-mental state Orientation, short-term 16 complex items: Elders, 
examination (MMSE) | memory registration, Qualitative and quantitative | potentially 
[136] attention, calculation, questions pathologic 
recall, language, and task 
reproduction 
Trail making test Visual search, scanning, Two items: The participant | Non- 
(TMT) [137] processing speed, mental | connects circles denoted by | pathological 
flexibility, and executive numbers and letters in 
functions. ascending order 
Selective reminding | Verbal memory One item: The participant | Non- 
test (SRT) [138] recalls as many as possible | pathological 
of 12 dictated unrelated 
words 
Spatial recall test Visuospatial learning, the | One item: The participant | Pathologic, 
(SPART) [139, 140] | susceptibility of such recalls as many as possible | multiple 
learning to proactive and of 10 checkers on a sclerosis 
retroactive interference, 36-checkers square board 
and the ability to recall 
visuospatial information 
following a period of delay 
Symbol digits Presence of organic One item: The participant | Pathologic, 
modalities test cerebral dysfunction has 90 seconds to pair cerebral 
(SDMT) [141] leading to neurological specific numbers with given | dysfunction 
impairment geometric figures. 
Paced auditory serial | Rate of information Multiple items: The Pathologic, 
addition test (PASAT) | processing after recovering | participant hears a series of | multiple 
[142] from trauma digits, one every 3 seconds, | sclerosis 
and reports the sum of the 
last two digits. 
Word list generation | Neuropsychological One item: The participant | Pathologic, 
(WLG) [143] measures of verbal fluency | generates words from a dementia, 
restricted category (e.g., multiple 
starting with S or denoting |sclerosis 
animals) in 60 seconds. 
Rey auditory verbal | Recent memory, verbal Multiple items: 15 nouns Non- 
learning test learning, susceptibility to | read aloud each second for | pathological 
(RAVLT) [69] interference, and retention |5 consecutive trials 
of information after a followed by participant 
certain period of time recall. 
during which other 
activities are performed 
Simple reaction time | Relationships between the | One item: A square is Non- 
task (SRTT) [144] deceleration of heart rate shown on screen at pathological 


Observed to anticipate both 
aversive and non-aversive 
stimuli, and several aspects 
of the somatic-motor 
activity. 


different intervals. The 
participant selects a button 
to react to seeing the 
square. 


(continued) 
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Task Measures Administration Usage 
Psychomotor Impact of loss of sleep Multiple items: Ranging up | Non- 
vigilance task (PVT) | sustained wakefulness, and/ | to 10 minutes, similar to pathological 
[23] or time of day on the SRTT. 

neurobehavioral 

performance 
Brief Repeatable Selective short-term Multiple tests: Selective Pathologic, 
Battery of memory, spatial recall, reminding test (SRT), multiple 
Neuropsychological | symbol digit modalities, spatial recall test (SPART), | sclerosis 


Tests [145] 


paced auditory serial 
addition, and word list 
generation; first used for 
multiple sclerosis 


symbol digits modalities 
test (SDMT), paced 
auditory serial addition test 
(PASAT), delayed recall of 
the SRT, delayed recall of 
the SPART, and word list 
generation (WLG). 


Fig. 4.3 Electrocardiogram. © Blausen Electrocardiogram by BruceBlaus licensed under CC- 


BY-3.0 [158] 
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Studies using the ECG to assess fatigue in a non-pathological, occupational con- 
text include airline crew [159], surgeons [160] or 3D TV watchers [161]. In these 
studies, the ECG was measured with electrode-based devices before and after the 
tiring task (i.e., via an indirect measurement approach). HRV pre- and post-task was 
used as a measure for fatigue in work settings, e.g., emergency and pre-hospital doc- 
tors [162]. 

Measurements of cardiac physiology have been performed during daily life (1.e., 
via a direct measurement approach) also in a non-pathological setting. A large body 
of research focused on assessing cardiac activity in healthcare and driving profes- 
sionals. Medical interns were given Holter recorders throughout the day, measuring 
HR and HRY, in conjunction with resting ECG to assess fatigue [163]. Surgeon 
HRV (using EEG) was assessed in robot-assisted versus conventional cholecystec- 
tomy [164]. Drivers were assessed while driving, through an ECG device mounted 
on the steering wheel [165]. Another study assessed the impact of electroacupunc- 
ture on fatigue and Quality of Life using subjective SF-36 and objective HRV using 
ECG (SphygmoCor) [166]. A method aimed at estimating the perception of physi- 
cal fatigue by predicting heart rate through smartphones has been proposed by esti- 
mating the oxygen consumption, using a smartphone acceleration and location (via 
accelerometer and GPS, respectively) [3]. The study yielded an adequate detection 
of fatigue when individuals performed daily-life activities under naturalistic 
conditions. 


Ocular Physiology 


Keeping the eye closed or having fixed changes in pupil diameter have been observed 
in a state of fatigue [167] due to monotony or sleep deprivation. Ocular physiology 
measures used for assessing fatigue include the spontaneous eye blink [168], pupil 
diameter [169], oscillations in pupil diameter (fatigue waves) [170, 171]. Another 
method used to detect fatigue is the electrooculogram (EOG), an electrophysiologi- 
cal method, which measures the resting electrical potential between the cornea and 
Bruch's membrane. 

Studies using ocular physiology measures were primarily done to assess fatigue 
in non-pathological, occupational settings, e.g., in the military detecting sleep 
deprivation-induced fatigue by saccade peak velocity in the Navy using question- 
naires (on PDAs), actigraphy (Actiwatch), and EOG (Natus, then Embla) during a 
saccade task [172] or assessing fatigue in the Air Force through saccadic velocity 
using software (Eyelink) in a dark room before and after a long flight [22]. For 
driver drowsiness, studies assessed fatigue by EOG using a device mounted next to 
the eyes for brief periods [21]. 
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Smartphones have been utilized and applied to drivers as well. Researchers have 
presented an app, which uses information from both front and back cameras and 
others embedded sensors on the phone to detect and alert drivers to dangerous driv- 
ing conditions inside and outside the car [173]. Researchers used computer vision 
and machine learning algorithms on the phone to monitor and detect whether the 
driver is tired or distracted using the front camera while at the same time tracking 
road conditions using the back camera. The front camera pipeline tracks the driver's 
head pose and direction as well as eyes and blinking rate as a means to infer drowsi- 
ness and distraction. Specifically, researchers used blink detection algorithms to 
detect periods of micro-sleep, fatigue and drowsiness. A more recent study improved 
EOG by mounting the device on the forehead to increase the duration of comfort- 
able measurement [174]. 


Neural Physiology 


Neural electrophysiological measures used to assess fatigue include the electroen- 
cephalogram (EEG), the evoked response potential (ERP), the Error Related 
Negativity, and lateralized readiness potential [16]. Magnetic resonance imaging 
(MRI) was also used to identify factors of fatigue [175]. This type of objective mea- 
sure focuses on cognitive performance, described in a preceding section, by requir- 
ing the participants to conduct a task while monitoring takes place. 

Studies have assessed neural physiology of fatigue in a non-pathological context 
by using EEG or ERP in the general population [176, 177], as well as EEG on occu- 
pational fatigue, e.g., drivers [178, 179], and surgeons while conducting a demand- 
ing task. For surgeons, Kahol [180] studied the impact of fatigue in surgical 
residents, which used a demanding task and measurement by EEG using a B-Alert 
device while Guru [181] assessed cognitive performance during robot-assisted sur- 
gery by EEG using a B-Alert device. Other studies which used electrophysiological 
measures in conjunction with other methods are elaborated on in the objective mea- 
sures section of mix methods. 


Biologic Markers 


Fatigue-related biologic markers were studied in the pathological context of chronic 
disease: plasma glucose, associated with variations in transient physical and mental 
energy, effort, and fatigue with variable degrees of success [182, 183]; cortisol, an 
indirect marker of fatigue through stress level and energy expenditure associated 
with fatigue [184]; salivary alpha-amylase (sAA) associated with surrogate markers 
of nervous system activity [185] and task engagement/disengagement [186], with 
variable degrees of success; and melatonin following circadian patterns and dis- 
rupted in individuals with chronic disease and recurrent fatigue [187], used for 
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sleep-related fatigue. In elite athletes, creatine kinase (CK), C-reactive protein 
(CRP), uric acid, testosterone, salivary immunoglobulin (S-IgA) were used as indi- 
rect markers of fatigue in the recovery period following intense physical activity. 
Biologic systems involved in the regulation of motor activity are intricately linked 
with sleep, feeding behaviour, energy, and mood [188]. 


Behavioural Markers 


Common behavioural markers utilized to assess fatigue include sleep and physical 
activity. These markers can be assessed by research-grade devices and consumer 
devices alike, with various degrees of validated accuracy, wear comfort, and pres- 
ence in the research lab for the procedure. Figures 4.4 and 4.5 depict research and 
consumer wearable devices, respectively. As opposed to the momentary measures 
above, the behavioural markers can also be monitored continuously (with very high 
frequency, e.g., seconds or milliseconds) and longitudinally (for an extended dura- 
tion, e.g., weeks to years) in time. 

Sleep can be assessed using polysomnography and actigraphy. Polysomnography 
(PSG) [189] is an electrophysiological sleep study, which assesses brain waves 
(EEG), oxygen levels in the blood, heart rate (ECG), eye movements (EOG), and 
muscle and skeletal muscle activation and movements (EMG), breathing functions, 
respiratory airflow, respiratory effort, and pulse oximetry (SpO;). Polysomnography 
quantifies sleep duration, interruptions, stages (e.g., light, deep, rapid eye move- 
ment (REM)) and waking states (e.g., awake, asleep). Actigraphy [190] is a non- 
invasive electrophysiological method that assesses movement and is used to monitor 
humans at rest or during various types of physical activity. Examples of 


Fig. 4.4 Research-grade 
wearable (pedometer). © 
Pedometer Take-Apart by 
llenore licensed under CC 
BY 2.0 
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Fig. 4.5 Consumer 
wearable (smartwatch). © 
Moto 360 Smartwatch by 
chrisf608 licensed under 
CC BY 2.0 


research-grade wearable actigraph devices are ActiWatch! and ActiGraph.? The 
actigraph can be worn on the wrist or ankle during daily life, for several weeks. The 
actigraph allows for the continuous collection of data due to its non-invasive nature, 
however, widespread and longitudinal use is limited by its specific purpose of 
researching physical activity with limited considerations to the user experience 
and price. 

More recent consumer wearable monitors, in the form of wristbands, smart- 
watches, sleep mattresses, or finger rings from manufacturers such as Fitbit,? Oura,* 
and Withings? [191] monitor sleep continuously by using a combination of move- 
ment, measured by a triaxial accelerometer, and HR/HRV, measured by photople- 
thysmography (PPG), non-invasive optical measurement of the volumetric 
variability of blood in the vessels under the skin. Consumer wearables can also 
measure behavioural markers pertaining to physical activity, e.g., duration, intensity 


! https://www.usa.philips.com/healthcare/product/HC1046964/actiwatch-spectrum- 
activity-monitor 

*https://www.actigraphcorp.com 

Shttps://fitbit.com 

*https://ouraring.com 

Shttps://withings.com 
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(classified as, e.g., sedentary, low, moderate, and vigorous), type (using activity 
class recognition), effort (in metabolic equivalent of tasks (METs)), distance, eleva- 
tion, step count, workouts, and other measures derived from the continuous multi- 
variate data obtained from triaxial accelerometer and gyroscope sensors inside 
the device. 

Studies assessing non-pathologic fatigue, sleep, and physical activity have been 
performed in segments of the general population and for several occupations, usu- 
ally by combining subjective and objective measurements. In segments of the gen- 
eral population, Ellingson [17] studied the influence of active and sedentary 
behaviours on perceived energy and fatigue in women by using subjective POMS 
and SF-36 and objective physical activity by an accelerometer (Actigraph). For 
occupations, Rizzo [192] assessed the role of fatigue and sleepiness in drivers with 
obstructive sleep apnea by using subjective SF-36 and objective PSG. De Aratijo 
Fernandes Jr. [193] quantified the impact of shift work on train drivers by using 
PVT and actigraphy (Actiwatch). Fernandes-Junior [193] assessed sleep, fatigue, 
and Quality of Life in night shift workers using subjective scale and actigraphy 
(Actiwatch). Towards the pathologic type of fatigue, Campbell [194] assessed 
fatigue and sleep in individuals having unexplained chronic fatigue by using subjec- 
tive scales and objective PSG; Maher [195] quantified the relationships between 
fatigue, physical activity, and socio-demographic characteristics in children and 
adolescents with physical disabilities by using objective physical activity measure- 
ment using an accelerometer (Actigraph). 

Numerous other studies have assessed pathologic fatigue in the context of a spe- 
cific disease using PSG or actigraphy. Attarian [196], Kaynak [197], Veauthier 
[198], and Kaminska [199] studied relationships between sleep and fatigue in mul- 
tiple sclerosis patients. Keefer (2006) and Shitrit [200] assessed sleep and fatigue in 
inflammatory bowel disease. Merikangas [188] used a combination of EMA and 
actigraphy to assess energy, mood, and activity in individuals with depressive disor- 
ders. Sun [201] assessed the relationships between daytime napping and fatigue and 
Quality of Life in cancer individuals by using subjective scale and objective sleep 
quality (Actigraph). Ancoli-Israel [202] assessed sleep, fatigue, and circadian activ- 
ity in women with breast cancer by using subjective scale and objective circadian 
rhythms using actigraphy (Actiwatch). Holliday [203] assessed fatigue and sleep 
quality in prostate cancer patients by using a subjective scale of Quality of Life and 
actigraphy (Actiwatch). Cambras [204] studied circadian rhythm in patients of 
encephalomyelitis using actigraphy (ActTrust). Nicklas [205] assessed physical 
activity behaviours (using accelerometers) and fatigue (using SF-36) in adults of 
middle and old age with chronic inflammations. Nilsson [206] studied intensity 
levels of physical activity and fatigue in cancer patients by using an accelerometer 
(SenseWear). Vancampfort [207] studied the relationships between cardio- 
respiratory fitness and increased quality of life in people with bipolar disorder using, 
among others, the subjective SF-36 and an armband (SenseWear) for objective 
physical activity and sedentary behaviour measurement. Sheshadri [208] assessed 
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the relationship between intensity levels of physical activity and fatigue in patients 
on dialysis by using step count from a pedometer (Accusplit). 

More recent studies used wearables to assess wearable-measured sleep and phys- 
ical activity in a pathologic context. Qazi [209] studied fatigue in patients with 
inflammatory bowel disease by using a Fitbit Charge HR. Sofia et al. [210] used the 
same wearable to associate sleep fragmentation with individuals having clinically 
active disease. Abbott [211] conducted an intervention study for physical activity in 
case of cancer-related fatigue patients by using activity trackers (undisclosed brand) 
without reporting measurements but reporting that the activity tracker was deemed 
helpful. 


Mixed Methods 


In our literature review, we identified numerous studies which combined two or 
more objective measures of fatigue. These studies focused on either cognitive or 
physical fatigue in the general population or specific occupations, or physical 
fatigue in specific segments of the population. 

For non-pathologic cognitive fatigue in the general population, Zhang [174] esti- 
mated mental fatigue based on EEG (Neuroscan) and HRV from ECG while per- 
forming an arithmetic task using a personal computer, Ren [212] studied various 
degrees of mental fatigue by using multiple types of measurements: EEG, ECG as 
well as galvanic skin response (GSR), Smith [213] quantified the effects on cogni- 
tive tasks on mental fatigue indicators, using PVT and other two tasks and assessing 
fatigue through subjective VAS and objective HRV from EEG, and Brown [214] 
studied the effects of mental fatigue on exercise intentions and behaviour using 
cognitive and then physical exercises by using a cycle ergometer. 

In the area of non-pathological physical fatigue, Kanitz [215] assessed the impact 
on eurythmy therapy on fatigue by using subjective MFI and objective HRV by 
ECG. For occupational fatigue, Smolders [216] studied the alertness during office 
hours induced by higher luminosity by using subjective measures, task performance 
(PVT, letter substitution test), and heart rate measures (ECG), Oriyama [217] stud- 
ied fatigue in shift nurses by measuring objective HRV from ECG, and subjective 
EMA using VAS, and Singh [218] assessed the technical performance of surgeons 
when using robotic surgery where the task was a suture under time pressure, mea- 
sured with a subjective surgical task scale and objective HR, and objective func- 
tional near-infrared spectroscopy (fNIRS). 

In the area of pathological fatigue, Dishman [219] studied the effects of cycling 
exercise on fatigue among young adults who report persistent fatigue using incre- 
mental exercise test on an electronically braked, computer-driven cycle ergometer 
(Lode), and providing subjective POMS and objective HR (Polar), VO;-max and 
expired gas (Parvo Medics), and EEG (Electrical Geodesics). 
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Table 4.4 Fatigue measurements and spectrums of characteristics from the literature review 


Measurement | Subjective Objective 
Both office Both office and 
Location and daily life | Daily life Office daily life Daily life 
Reporting Self-reported Perf-reported | Tech-reported 
Administration | Scales Prompts, Task hardware | Research Consumer 
e.g., EMA and devices devices devices 
Validated Yes Partial Yes Yes Partial 
Quantifiable No No Yes Yes Yes 
Frequent No Yes No No Yes 
Continuous No No Yes Yes Yes 
Judgment-free | No No Partial Yes Yes 
Mood-free No No Yes Yes Yes 
Memory-free No Partial Yes Yes Yes 
Owned Partial Partial No No Yes 
Contextual No Yes No Partial Yes 
Property Spectrums of Energy and Fatigue Measures 


The findings from our literature review classify the energy and fatigue measure- 
ments by type (subjective and objective), location (clinician's office, daily life, or 
both/mixed), source (self-, performance/capacity-, and technology-reported, using 
the taxonomy by Mayo [12]), and administration (scales, prompts, tasks, and 
devices). In Table 4.4, we place each such measurement on spectrums for the fol- 
lowing properties: 


1. 


2. 
. Frequent: often repeated administrations with one day or less between 


Validated: fatigue outcome reliability assessed by statistical analysis on the tar- 
get population and scientific publication. 
Quantifiable: fatigue outcomes interval or ratio at a minute or higher precision. 


administrations. 
Continuous: fatigue proxy variable measured on a time series with a minute or 
higher granularity. 


. Judgment-free: bias-free from the perception of judgment from the administra- 


tor; tasks and research devices allow some refraining. 
Mood-free: bias-free from the voluntary or involuntary perception of self. 


. Memory-free: bias-free from the remembrance of the past; prompts allow for 


long-term memory loss. 


. Owned: whether the participant owns the device; scales and prompts are marked 


as partial in case they are delivered to a device owned by the participant. 


. Contextual: collected from settings daily life; research devices can be borrowed 


to the participant for a short time to wear in daily life context. 
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Discussion 
Key Findings 


Fatigue or lack of energy is a universal symptom experienced by those suffering from 
different medical and psychological illnesses as well as by healthy individuals in the 
general population. Overall, fatigue is a ubiquitous and multifaceted symptom that is 
challenging to define and measure. Fatigue may be classified as pathological or non- 
pathological, physical or mental, and can be measured subjectively or objectively. 

Different approaches have been employed in order to measure energy and fatigue 
including scales, prompts, physical measures, cognitive measures, physiological 
markers, biological markers, behavioural markers, and mixed methods. Some mea- 
surement methods assess the effects of fatigue (e.g. performance decrements), some 
attempt to identify the source of fatigue (e.g. muscle dysfunction), while others 
adopt a behavioural perspective (e.g. decreased physical activity or prolonged 
sleep). Some methods focus on capacity while others assess performance. These 
varied methods each contain advantages and disadvantages in terms of traditional 
validation, access to continuous data, and ecological validity. 

Subjective instruments instantiating self-reported outcomes [12] suffer from 
inherent shortcomings, in particular, they are infrequent and subjective. Furthermore, 
self-report by recall has an intrinsic problem: due to biases, such as mood states or 
sleepiness, individuals are not able to accurately recall past experience, particularly 
experiences that are frequent, mundane, and/or irregular [220]. In addition, the 
potential discrepancy between how one feels and how one thinks one should feel 
contributes to lack of ecological validity in self-reports of fatigue and requires fur- 
ther research [15]. Incorporating a real-time collection of fatigue data in naturalistic 
settings may reduce problems associated with retrospective recall of events, sum- 
marization of events, and artificial contexts or settings [118]. 

Objective measures obtained by tech-reported outcomes can be collected con- 
tinuously from individuals in the context of daily life. To this end, both academia 
and industry are increasing their efforts to develop technological solutions, such as 
sensors which can measure, models which can assess, and artefacts which can man- 
age energy and fatigue. Recent technological methods to monitor and manage 
energy and fatigue include sensors, smartphones and their applications, and 
research- and consumer-grade wearables. Technology-based monitoring of energy 
and fatigue could assist in the initial diagnosis and the early detection of diseases 
could enable one to monitor post-treatment evolution and could help assess the risk 
of certain medications on patients [3]. Furthermore, technology-based monitoring 
of energy and fatigue could assist healthy individuals in enhancing work perfor- 
mance, conserving and managing energy levels, and maintaining health. 

Energy and fatigue are of great importance to diseased individuals. The connec- 
tion between pathological fatigue and disease is well established in the literature. 
Fatigue frequently foreshadows conditions like multiple sclerosis [221], cancer 
[28], and HIV infection [222], among other diseases. Furthermore, fatigue, as well 
as increased energy, has been identified as a core symptom of mental health disor- 
ders including depressive disorders and bipolar disorders. Current literature on 
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energy and fatigue is biased towards pathological, rather than healthy, populations. 
In addition, it is possible that the existence of healthy fatigue is a barrier to full 
comprehension of the impact of pathological fatigue, as pathological fatigue is more 
extreme and different. This highlights the importance of further research on both 
non pathological and pathological fatigue. 

In addition to the comprehensive literature examining fatigue and disease, the 
monitoring of energy and fatigue has also been highlighted for specific vocational 
and occupational populations, such as professional athletes [24], police [25], and 
drivers [165]. The literature aims to gain an understanding of health, safety, occupa- 
tional functioning, burnout, performance, and capacity. More efforts could be put 
toward studying healthy general populations, as in addition to affecting an individ- 
ual’s quality of life, fatigue impacts the economy because of the connection to pro- 
ductivity and illness. 

Insights into the classification and measurement of energy and fatigue may also 
be applied broadly to the general population as mobile monitoring technology 
allows the assessment of these homeostatic systems in real-time [188]. Quality of 
Life Technologies (QoLT) refers to technologies for assessment or improvement of 
the individual’s quality of life [223]. Optimal measurement of energy and fatigue 
would be moved out of the lab and into the real world, continuous rather than infre- 
quent, and based on accurate, validated, yet minimally intrusive measures and 
devices. Future research could establish traditional validity for the continuous, daily 
life, measurement of energy and fatigue. 

Assessing energy and fatigue could also contribute to the quantified self. The 
quantified self (QS) is any individual engaged in the self-tracking of any kind of 
biological, physical, behavioural, or environmental information. QS promotes a 
proactive stance toward obtaining information and acting on it [224]. One of the 
earliest recorded examples of quantified self-tracking is that of Sanctorius of Padua, 
who studied energy expenditure by tracking his food intake, weight, and elimination 
for 30 years in the sixteenth century [225]. State of the art energy and fatigue assess- 
ment could contribute meaningfully to the quantified self. 


Limitations 


A limitation of the current chapter stems from the pathological bias in the field. 
Namely, because the existing literature is biased toward pathological fatigue, we 
built the non-pathological (also referred to as physiological) classification system 
arm based on existing pathological models. This limitation is also related to our 
literature search strategy. Our method of reviewing the literature was based on a 
scoping review approach rather than a structured systematic review. We did not 
exclude studies based on methodologies used or populations studied. 

Subjective measures discussed in this chapter contain limitations including being 
infrequent, involving recalls, and potential to be influenced by mood states, mem- 
ory, and expectations. Wearable measurements also contain limitations related to 
the population that uses wearables. Specifically, device owners are more likely to be 
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young individuals with disposable incomes who already lead healthy lifestyles and 
want to quantify their progress [226]. Future work should ensure that wearable data 
is representative and note this bias in current wearable data. 

An additional limitation of the field is that there is not yet a validated calibration 
between objective measures and the concept of energy and fatigue. Therefore, much of 
our discussion is speculative. A major impediment in the understanding of fatigue and 
energy lies in the fact that for over 100 years, research has shown little relationship 
between self-report and actual, objective measurements of fatigue [167]. There are sev- 
eral definitions of energy and fatigue and these have not been conclusively associated 
with objective measures. This doesn't invalidate subjective or objective measures of 
fatigue but rather indicates that they may be describing something that is more compli- 
cated and cannot be whittled down to a single biological measure. Therefore, both sub- 
jective experience and objective measurements are being considered in the context of 
energy and fatigue, as they are important indicators for health and quality of life. Future 
research could aim to bridge the gap between subjective and objective measures by 
accounting for multiple variables and conducting calibration studies. 


Opportunities 


Energy and fatigue is a Quality of Life facet in which the successful assessment, 
exclusively through Quality of Life Technologies [223], has promising likelihood. 
The mass adoption of miniaturized devices in daily life (with large scale and diver- 
sity in personal and contextual characteristics of the data), the availability of rele- 
vant predictors of energy and fatigue in large scale data, and the presence of 
platforms that facilitate participation in research at scale contribute to the feasibility 
of the operationalization of this facet. 

Currently, research is progressing in assessing pathological and non-pathological 
energy and fatigue by using subjective, objective, and mixed methods. Miniaturized 
devices, such as smartphones and wearables, increasingly accurately monitor daily 
life behaviours (e.g., physical activity and sleep), sense signals (e.g., heart rate, 
momentary electrocardiogram, etc.) and administer prompts (e.g., validated scales, 
items, and tasks). As the line between consumer health wearables and medical devices 
continues to blur, it is possible for a single wearable device to monitor a range of 
medical risk factors [227]. Adoption of wearables is increasing; 2196 of Americans 
own a wearable [228], there are more than 200 models of wearables? and the market 
is expected to continue to increase by 2022 [229] towards available objective behav- 
ioural data at scale. Open health platforms are being employed to facilitate scalable 
participation and manage subjective, objective, and mixed data [230]. 

Co-calibrations of (1) subjective validated scales of energy and fatigue and (2) 
objective measures of daily life behaviours may rigorously validate objective mea- 
sures of energy and fatigue and meet the aim of assessing energy and fatigue using 
QoLT. For example, a study aiming to co-calibrate subjective scales and objective 
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behaviours for occupational fatigue may collect multiple behavioural markers pas- 
sively and continuously (e.g., physical activity, sleep, heart rate) from tens to hun- 
dreds of drivers for several months to years, during driving and daily living, and 
regularly administering validated energy and fatigue scales such that their recall 
periods cover the duration. Such a study may observe trends of fatigue longitudi- 
nally in time. Within a smaller sample size, a purely statistical approach would 
allow for the assessment of validity (e.g., by correlating the corresponding subjec- 
tive and objective measures) and reliability (e.g., by measuring the same person’s 
fatigue in similar days of week, months, or seasons) of the objective measure. 
Within a larger sample, a predictive approach would learn the subjective measures 
of energy and fatigue by using the objective measures of behaviours. These 
approaches can iteratively reduce the number of scale items. One step further, con- 
tinuous behaviour monitoring during daily life facilitates the trigger of momentary 
assessments upon changes in objective behaviours that associate with changes in 
energy and fatigue. Such an approach may increase the accuracy of the co- 
calibration. Furthermore, alternative statistical or predictive risk scenarios can 
maintain energy (“if you continue working at this pace, you will likely not get 
tired"), prevent fatigue (“if you continue working at this pace, you will likely accu- 
mulate occupational fatigue in two weeks"), and compensate for the losses induced 
by fatigue ("consider taking a break of one week to restore your productivity from 
three months ago"). 

Initially, co-calibrations may suffer from lower accuracy (e.g., revealing only 
basic trends and associations) or limited extent (e.g., applying for specific scale 
items, collecting limited objective behaviours, applying for limited energy and 
fatigue types) as the measured objective measures or available sample may not 
explain the energy and fatigue directly. In such cases, a directed graph of co- 
calibrations with additional Quality of Life facets (e.g., stress, health outcomes), 
using additional objective measures, may need to be constructed to represent the 
relationships accurately such that energy and fatigue are explained through a series 
of directed co-calibration paths originating exclusively from objective measures, 
essentially assessing energy and fatigue through QoLT exclusively. 

A successful energy and fatigue assessment using QoLT would contribute to the 
"Internet of everything" 50-year vision of a digital future where "internet use will 
be nearly as pervasive and necessary as oxygen" [231]. Specifically, such an assess- 
ment would contribute to three of Stansberry's five hopeful visions of 2069. The 
first vision, living longer and feeling better where "internet-enabled technology will 
help people live longer and healthier lives; scientific advances will continue to blur 
the line between human and machine" [231] will be enabled by quantifying the 
relationships between energy, fatigue, behaviours, health, and Quality of Life out- 
comes. The second vision, less work, more leisure where "artificial intelligence 
tools will take over repetitive, unsafe and physically taxing labour, leaving humans 
with more time for leisure" [231] will be enabled through (short-term) the transition 
to increasingly passive reported outcomes that reduce the burden of participation in 
research and (longer-term) statistical and predictive optimization of physical and 
mental effort allocation for the occupations where energy and fatigue are prevalent. 
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The third vision, individualized experiences where “digital life will be tailored to 
each user” [230] will be enabled by interventions leveraging large scale data, accu- 
rate models, and alternative personalized scenarios addressing fatigue prevention, 
before management, and before compensation. 


Conclusive Remarks 


Energy and fatigue impact physical, cognitive, emotional, social, and occupational 
functioning and carry important implications for an individual’s health and overall 
Quality of Life. Lacking energy carries consequences for an individual’s routine func- 
tioning. Everyday activities, including work performance and self-care activities, can 
be impeded or even curtailed. Energy is required to sustain life and efficient spending 
of energy results in overall vitality. Paradoxically, one tends to think of energy of a 
resource that is depleted and then restored with rest, while at the same time, many 
observe that using energy generates additional energy. This curious paradox high- 
lights the importance of future research and clarity. In addition, the classification of 
energy and fatigue is critical as it is possible that the existence of non pathological 
fatigue inhibits true appreciation of the impact of pathological fatigue. 

The contributions of this chapter include a semi-structured literature review on 
energy and fatigue assessment and its potential within Quality of Life Technologies, 
a taxonomy of the field of energy and fatigue, and the identification of a research 
validation gap between subjective and objective measures of energy and fatigue. We 
foresee the necessity to conduct studies of increasing size in order to co-calibrate 
the subjective and objective measures towards the integration of exclusively objec- 
tive measures in research and clinical practice. 

The measurement of energy and fatigue has been complicated by difficulties in 
definition and assessment. We conclude that optimal classification and measure- 
ment of energy and fatigue would occur in the real world, continuously and in real- 
time, while being ecologically valid and informing the design of interventions 
aimed at maintaining energy and monitoring fatigue towards positive outcomes of 
health and Quality of Life. 


References 


1. “WHO | WHOQOL: Measuring quality of life’ WHO. https://www.who.int/healthinfo/sur- 
vey/whogol-qualityoflife/en/index4.html. Accessed Jun. 24, 2020. 

2. Stahl SM. The psychopharmacology of energy and fatigue. J Clin Psychiatry. 2002, 
Jan;63(1):7-8. https://doi.org/10.4088/jcp.v63n0102. 

3. Hernandez N, Favela J. Assessing the perception of physical fatigue using Mobile sensing. 
In: Wister M, Pancardo P, Acosta F, Hernandez JA, editors. Intelligent data sensing and pro- 
cessing for health and Well-being applications. Academic; 2018. p. 161-73. 

4. Boksem MAS, Tops M. Mental fatigue: costs and benefits. Brain Res Rev. 2008, 
Nov;59(1):125—39. https://doi.org/10.1016/j.brainresrev.2008.07.001. 


4 Quantifying Energy and Fatigue: Classification and Assessment of Energy and... 107 


10. 


11. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 


19. 


20. 


21. 


22. 


23. 


24. 


. Marcora SM, Staiano W, Manning V. Mental fatigue impairs physical performance in humans. 


J Appl Physiol. 2009, Mar;106(3):857—64. https://doi.org/10.1152/japplphysiol.91324.2008. 


. Al-Mulla MR, Sepulveda F, Colley M. A review of non-invasive techniques to detect and 


predict localised muscle fatigue. Sensors. 2011;11(4):4. https://doi.org/10.3390/s 110403545. 


. Finsterer J, Mahjoub SZ. Fatigue in healthy and diseased individuals. Am J Hosp Palliat Med. 


2014, Aug;31(5):562-75. https://doi.org/10.1177/10499091 13494748. 


. Kluger BM, Krupp LB, Enoka RM. Fatigue and fatigability in neurologic illnesses. 


Neurology. 2013, Jan;80(4):409-16. https://doi.org/10.1212/WNL.0b013e31827f07be. 


. Chaudhuri A, Behan PO. Fatigue in neurological disorders. Lancet. 2004;363(9413):978-88. 


https://doi.org/10.1016/S0140-6736(04)15794-2. 

Mollayeva T, Kendzerska T, Mollayeva S, Shapiro CM, Colantonio A, Cassidy JD. A sys- 
tematic review of fatigue in patients with traumatic brain injury: the course, predictors and 
consequences. Neurosci Biobehav Rev. 2014, Nov;47:684—716. https://doi.org/10.1016/j. 
neubiorev.2014.10.024. 

Ream E, Richardson A. Fatigue: a concept analysis. Int J Nurs Stud. 1996, Oct;33(5):519-29. 
https://doi.org/10.1016/0020-7489(96)00004-1. 

Mayo NE, Figueiredo S, Ahmed S, Bartlett SJ. Montreal accord on patient-reported out- 
comes (PROs) use series-paper 2: terminology proposed to measure what matters in health. 
J Clin Epidemiol. 2017, Sep;89:119—24. https://doi.org/10.1016/j.jclinepi.2017.04.013. 
Dittner AJ, Wessely SC, Brown RG. The assessment of fatigue: a practical guide for clini- 
cians and researchers. J Psychosom Res. 2004, Feb;56(2):157—70. https://doi.org/10.1016/ 
$0022-3999(03)0037 1-4. 

Whitehead L. The measurement of fatigue in chronic illness: a systematic review of uni- 
dimensional and multidimensional fatigue measures. J Pain Symptom Manag. 2009, 
Jan;37(1):107—28. https://doi.org/10.1016/j.jpainsymman.2007.08.019. 

Aaronson LS, et al. Defining and measuring fatigue. Image J Nurs Sch. 1999;31(1):45—50. 
https://doi.org/10.1111/j.1547-5069.1999.tb00420.x. 

Hornsby BWY, Naylor G, Bess FH. A taxonomy of fatigue concepts and their rela- 
tion to hearing loss. Ear Hear. 2016;37(Suppl 1):136S8-44S. https://doi.org/10.1097/ 
AUD.0000000000000289. 

Ellingson L, Ae K, Vack N, Cook D. Active and sedentary behaviors influence feelings of 
energy and fatigue in women. Med Sci Sports Exerc. 2014, Jan;46(1):192—200. https://doi. 
org/10.1249/mss.0b013e3182a036ab. 

Abdel-Kader K, et al. Ecological momentary assessment of fatigue, sleepiness, and exhaus- 
tion in ESKD. BMC Nephrol. 2014, Feb;15(1):29. https://doi.org/10.1186/1471-2369-15-29. 
Cochran A, Belman-Wells L, McInnis M. Engagement strategies for self-monitoring symp- 
toms of bipolar disorder with Mobile and wearable technology: protocol for a randomized 
controlled trial. JMIR Res Protoc. 2018;7(5):e130. https://doi.org/10.2196/resprot.9899. 
Curran SL, Beacham AO, Andrykowski MA. Ecological momentary assessment of fatigue 
following breast cancer treatment. J Behav Med. 2004, Oct;27(5):425—-44. https://doi. 
org/10.1023/B:JOBM.0000047608.03692.0c. 

Chieh TC, Mohd M, Mustafa AH, Hendi SF, Majlis BY. Development of vehicle driver 
drowsiness detection system using electrooculogram (EOG). In: 2005 Ist International 
Conference on Computers, Communications, Signal Processing with Special Track on 
Biomedical Engineering; 2005, Nov. p. 165-8. https://doi.org/10.1109/CCSP.2005.4977181. 
Diaz-Piedra C, Rieiro H, Suárez J, Rios-Tejada F, Catena A, Stasi LLD. Fatigue in the mili- 
tary: towards a fatigue detection test based on the saccadic velocity. Physiol Meas. 2016, 
Aug;37(9):N62-75. https://doi.org/10.1088/0967-3334/37/9/N62. 

Dinges DF, Powell JW. Microcomputer analyses of performance on a portable, simple 
visual RT task during sustained operations. Behav Res Methods Instrum Comput. 1985, 
Nov;17(6):652—5. https://doi.org/10.3758/BF03200977. 

Thorpe RT, Atkinson G, Drust B, Gregson W. Monitoring fatigue status in elite team-sport 
athletes: implications for practice. Int J Sports Physiol Perform. Apr, 2017;12(s2):S2-27-34. 
https://doi.org/10.1123/ijspp.2016-0434. 


108 


25 


26. 


27. 


28. 


29. 


30. 


3L. 


32. 


33. 


34. 


35. 


36. 


3d. 


38. 


39. 


40. 


41. 


42. 
43. 


44. 


45. 


46. 


N. L. Solomon and V. Manea 


. Vila, Bryan, “Tired cops: the importance of managing police fatigue,’ Police Exec Res 
Forum, 2000. 

Glaus A. Fatigue in patients with cancer: analysis and assessment. Springer Science & 
Business Media; 2012. 

Jason LA, Evans M, Brown M, Porter N. What is fatigue? Pathological and nonpathological 
fatigue. PM&R. 2010, May;2(5):327-31. https://doi.org/10.1016/j.pmrj.2010.03.028. 

Davis MP, Khoshknabi D, Yue GH. Management of fatigue in cancer patients. Curr Pain 
Headache Rep. 2006; 10(4):260-9. https://doi.org/10.1007/s11916-006-0030-2. 

Tanaka M, Watanabe Y. Supraspinal regulation of physical fatigue. Neurosci Biobehav Rev. 
2012, Jan;36(1):727—34. https://doi.org/10.1016/j.neubiorev.2011.10.004. 

Bogdanis GCP. Effects of physical activity and inactivity on muscle fatigue. Front Physiol. 
2012;3 https://doi.org/10.3389/fphys.2012.00142. 

Bloomfield SA. Changes in musculoskeletal structure and function with prolonged bed rest. 
Med Sci Sports Exerc. 1997;29(2):197—206. 

Rimmer JH, Schiller W, Chen M-D. Effects of disability-associated low energy expendi- 
ture deconditioning syndrome. Exerc Sport Sci Rev. 2012, Jan;40(1):22-9. https://doi. 
org/10.1097/JES.0b013e31823b8b82. 

Dantzer R, et al. Identification and treatment of symptoms associated with inflammation 
in medically ill patients. Psychoneuroendocrinology. 2008, Jan;33(1):18—29. https://doi. 
org/10.1016/j.psyneuen.2007.10.008. 

Linnhoff S, Fiene M, Heinze H-J, Zaehle T. Cognitive fatigue in multiple sclerosis: An objec- 
tive approach to diagnosis and treatment by transcranial electrical stimulation. Brain Sci. 
2019, May;9(5):5. https://doi.org/10.3390/brainsci9050100. 

Valentine AD, Meyers CA. Cognitive and mood disturbance as causes and symptoms of 
fatigue in cancer patients. Cancer. 2001;92(S6):1694—8. https://doi.org/10.1002/1097-014 
2(20010915)92:6+<1694::AID-CNCR1499>3.0.CO;2-S. 

Kohl AD, Wylie GR, Genova HM, Hillary FG, DeLuca J. The neural correlates of cognitive 
fatigue in traumatic brain injury using functional MRI. Brain Inj. 2009, Jan;23(5):420-32. 
https://doi.org/10.1080/026990509027885 19. 

Byun E, Gay CL, Lee KA. Sleep, fatigue, and problems with cognitive function in adults liv- 
ing with HIV. J Assoc Nurses AIDS Care JANAC. 2016;27(1):5-16. https://doi.org/10.1016/j. 
jana.2015.10.002. 

Wong WS, Fielding R. Prevalence of chronic fatigue among Chinese adults in Hong Kong: a 
population-based study. J Affect Disord. 2010, Dec;127(1):248—56. https://doi.org/10.1016/j. 
jad.2010.04.029. 

Hockey B, Hockey R. The psychology of fatigue: work, effort and control. Cambridge 
University Press; 2013. 

Light AR, White AT, Hughen RW, Light KC. Moderate exercise increases expression for sen- 
sory, adrenergic and immune genes in chronic fatigue syndrome patients, but not in normal 
subjects. J Pain Off J Am Pain Soc. 2009, Oct;10(10):1099—112. https://doi.org/10.1016/j. 
jpain.2009.06.003. 

Schwartz JE, Jandorf L, Krupp LB. The measurement of fatigue: a new instrument. J 
Psychosom Res. 1993, Oct;37(7):753—62. https://doi.org/10.1016/0022-3999(93)90104-N. 
Edwards RH. Human muscle function and fatigue. Ciba Found Symp. 1981;82:1-8. 
Friedman JH, et al. Fatigue in Parkinson's disease: a review. Mov Disord. 2007;22(3):297-308. 
https://doi.org/10.1002/mds.21240. 

Cairns SP, Knicker AJ, Thompson MW, Sjggaard G. Evaluation of models used to study 
neuromuscular fatigue. Exerc Sport Sci Rev. 2005, Jan;33(1):9-16. 

Abd-Elfattah HM, Abdelazeim FH, Elshennawy S. Physical and cognitive conse- 
quences of fatigue: a review. J Adv Res. 2015, May;6(3):351-8. https://doi.org/10.1016/j. 
jare.2015.01.011. 

Kamijo K, Nishihira Y, Higashiura T, Kuroiwa K. The interactive effect of exercise 
intensity and task difficulty on human cognitive processing. Int J Psychophysiol. 2007, 
Aug;65(2):114—21. https://doi.org/10.1016/j.ijpsycho.2007.04.001. 


4 


47 


48. 


49. 


50. 


51. 


52. 


53. 


54. 


55. 


56. 


57. 


58. 


59. 


60. 


61. 


62. 


63. 


64. 


65. 


66. 


67. 


Quantifying Energy and Fatigue: Classification and Assessment of Energy and... 109 


. Tomporowski PD. Effects of acute bouts of exercise on cognition. Acta Psychol. 2003, 
Mar;112(3):297—324. https://doi.org/10.1016/80001-6918(02)00134-8. 

Cian C, Barraud PA, Melin B, Raphel C. Effects of fluid ingestion on cognitive function after 
heat stress or exercise-induced dehydration. Int J Psychophysiol. 2001, Nov;42(3):243—51. 
https://doi.org/10.1016/S0167-8760(01)00142-8. 

Falup-Pecurariu C. Fatigue assessment of Parkinson's disease patient in clinic: spe- 
cific versus holistic. J Neural Transm. 2013, Apr;120(4):577-81. https://doi.org/10.1007/ 
s00702-013-0969-1. 

Persson J, Welsh KM, Jonides J, Reuter-Lorenz PA. Cognitive fatigue of executive pro- 
cesses: interaction between interference resolution tasks. Neuropsychologia. 2007, 
Apr;45(7):1571-9. https://doi.org/10.1016/j.neuropsychologia.2006. 12.007. 

van der Linden D, Eling P. Mental fatigue disturbs local processing more than global process- 
ing. Psychol Res. 2006, Sep;70(5):395—402. https://doi.org/10.1007/s00426-005-0228-7. 
Lorist MM. Impact of top-down control during mental fatigue. Brain Res. 2008, 
Sep; 1232:113-23. https://doi.org/10.1016/j.brainres.2008.07.053. 

Cook DB, O’Connor PJ, Lange G, Steffener J. Functional neuroimaging correlates 
of mental fatigue induced by cognition among chronic fatigue syndrome patients 
and controls. | Neurolmage. 2007,May;36(1):108—22. _ https://doi.org/10.1016/j. 
neuroimage.2007.02.033. 

Lal SKL, Craig A. A critical review of the psychophysiology of driver fatigue. Biol Psychol. 
2001,Feb;55(3):173-94. https://doi.org/10.1016/S0301-0511(00)00085-5. 

Goode JH. Are pilots at risk of accidents due to fatigue? J Saf Res. 2003, Aug;34(3):309-13. 
https://doi.org/10.1016/S0022-4375(03)00033-1. 

Van Cutsem J, Marcora S, De Pauw K, Bailey S, Meeusen R, Roelands B. The effects of mental 
fatigue on physical performance: a systematic review. Sports Med. 2017, Aug;47(8): 1569-88. 
https://doi.org/10.1007/s40279-016-0672-0. 

Grillon C, Quispe-Escudero D, Mathur A, Ernst M. Mental fatigue impairs emotion regula- 
tion. Emot Wash DC. 2015, Jun;15(3):383-9. https://doi.org/10.1037/emo0000058. 

Chen MK. The epidemiology of self-perceived fatigue among adults. Prev Med. 1986, 
Jan;15(1):74—81. https://doi.org/10.1016/0091-7435(86)90037-X. 

Hardy GE, Shapiro DA, Borrill CS. Fatigue in the workforce of national health service 
trusts: levels of symptomatology and links with minor psychiatric disorder, demographic, 
occupational and work role factors. J Psychosom Res. 1997, Jul;43(1):83—92. https://doi. 
org/10.1016/S0022-3999(97)00019-6. 

Hickie IB, Hooker AW, Bennett BK, Hadzi-Pavlovic D, Wilson AJ, Lloyd AR. Fatigue in 
selected primary care settings: sociodemographic and psychiatric correlates. Med J Aust. 
1996; 164(10):585-8. https://doi.org/10.5694/).1326-5377.1996.tb122199.x. 

Loge JH, Ekeberg @, Kaasa S. Fatigue in the general norwegian population: normative 
data and associations. J Psychosom Res. 1998, Jul;45(1):53-65. https://doi.org/10.1016/ 
S0022-3999(97)0029 1-2. 

Pawlikowska T, Chalder T, Hirsch SR, Wallace P, Wright DJM, Wessely SC. Population 
based study of fatigue and psychological distress. BMJ. 1994, Mar;308(6931):763—6. https:// 
doi.org/10.1136/bmj.308.6931.763. 

David A, et al. Tired, weak, or in need of rest: fatigue among general practice attenders. 
BMJ. 1990, Nov;301(6762):1199-202. 

Bultmann U, Kant I, Kasl SV. Fatigue and psychological distress in the working population 
psychometrics, prevalence, and correlates. U Bu. 2002:8. 

Van Mens-Verhulst J, Bensing J. Distinguishing between chronic and nonchronic fatigue, 
the role of gender and age. Soc Sci Med. 1998, Sep;47(5):621—34. https://doi.org/10.1016/ 
S$0277-9536(98)001 16-6. 

Dement WC, Carskadon MA. Current perspectives on daytime sleepiness: the issues. Sleep. 
1982, Sep;5(suppl_2):S56—66. https://doi.org/10.1093/sleep/5.S2.S56. 

Shahid A, Shen J, Shapiro CM. Measurements of sleepiness and fatigue. J Psychosom Res. 
2010, Jul;69(1):8 1-9. https://doi.org/10.1016/j.jpsychores.2010.04.001. 


110 


68 


69. 
70. 


TT. 


72. 


T3: 


74. 


75. 


76. 


TI. 


78. 


79. 


80. 


81. 


82. 


83. 


84. 


85. 


86. 


87. 


N. L. Solomon and V. Manea 


. Dijk D, Czeisler C. Contribution of the circadian pacemaker and the sleep homeostat to 
sleep propensity, sleep structure, electroencephalographic slow waves, and sleep spin- 
dle activity in humans. J Neurosci. 1995, May;15(5):3526—38. https://doi.org/10.1523/ 
JNEUROSCI.15-05-03526.1995. 

Rey A. "L'examen clinique en psychologie." 1958. 

Michielsen HJ, De Vries J, Van Heck GL. Psychometric qualities of a brief self-rated fatigue 
measure: the fatigue assessment scale. J Psychosom Res. 2003, Apr;54(4):345—52. https:// 
doi.org/10.1016/S0022-3999(02)00392-6. 

Webster K, Odom L, Peterman A, Lent L, Cella D. The functional assessment of chronic ill- 
ness therapy (FACIT) measurement system: validation of version 4 of the core questionnaire. 
Qual Life Res. 1999; 

Fisk JD, Ritvo PG, Ross L, Haase DA, Marrie TJ, Schlech WF. Measuring the functional 
impact of fatigue: initial validation of the fatigue impact scale. Clin Infect Dis. 1994, 
Jan;18(Suppl_1):S79-83. https://doi.org/10.1093/clinids/18.Supplement_1.S79. 

Chalder T, et al. Development of a fatigue scale. J Psychosom Res. 1993, Feb;37(2):147—53. 
https://doi.org/10.1016/0022-3999(93)9008 1-P. 

Krupp LB, LaRocca NG, Muir-Nash J, Steinberg AD. The fatigue severity scale: application 
to patients with multiple sclerosis and systemic lupus erythematosus. Arch Neurol. 1989, 
Oct;46(10):1121-3. https://doi.org/10.1001/archneur.1989.005204601 15022. 

Tack BB. “Dimensions and correlates of fatigue in older adults with rheumatoid arthritis.,” 
Accessed: Jun. 24, 2020. [Online]. Available: https://elibrary.ru/item.asp?id=5833962. 
Smets EMA, Garssen B, Bonke B, De Haes JCJM. The multidimensional fatigue inventory 
(MFI) psychometric qualities of an instrument to assess fatigue. J Psychosom Res. 1995, 
Apr;39(3):3 15-25. https://doi.org/10.1016/0022-3999(94)00125-O. 

Ware JE Jr, Sherbourne CD. The MOS 36-item short-form health survey (SF-36): I. concep- 
tual framework and item selection. Med Care. 1992:473-83. 

Lai J-S, et al. How item banks and their application can influence measurement practice in 
rehabilitation medicine: a PROMIS fatigue item Bank example. Arch Phys Med Rehabil. 
2011, Oct;92(10):S20-7. https://doi.org/10.1016/j.apmr.2010.08.033. 

McNair D, Lorr M, Droppleman L. POMS manual for the profile of mood states. Educ Ind 
Test Serv. 1971; 

Lee KA, Hicks G, Nino-Murcia G. Validity and reliability of a scale to assess fatigue. 
Psychiatry Res. 1991, | Mar;36(3):291-8. _ https://doi.org/10.1016/0165-1781(91) 
90027-M. 

Junghaenel DU, Christodoulou C, Lai J-S, Stone AA. Demographic correlates of fatigue in 
the US general population: results from the patient-reported outcomes measurement infor- 
mation system (PROMIS) initiative. J Psychosom Res. 2011, Sep;71(3):117—23. https://doi. 
org/10.1016/j.jpsychores.2011.04.007. 

Alemohammad ZB, Sadeghniiat-Haghighi K. Risk of fatigue at work. In: Sharafkhaneh A, 
Hirshkowitz M, editors. Fatigue management: principles and practices for improving work- 
place safety. New York, NY: Springer; 2018. p. 181—91. 

Sharafkhaneh A, Hirshkowitz M, editors. Fatigue management: principles and practices for 
improving workplace safety. New York, NY: Springer New York; 2018. 

Xianglong S, Hu Z, Shumin F, Zhenning L. Bus drivers’ mood states and reaction abilities at 
high temperatures. Transp. Res. Part F Traffic Psychol. Behav. 2018, Nov;59:436-44. https:// 
doi.org/10.1016/,.trf.2018.09.022. 

Mitchell SA. Cancer-related fatigue: state of the science. PM&R. 2010, May;2(5):364—83. 
https://doi.org/10.1016/j.pmrj.2010.03.024. 

Pearson EJM, Morris ME, di Stefano M, McKinstry CE. Interventions for cancer-related 
fatigue: a scoping review. Eur J Cancer Care (Engl). 2018, Jan;27(1):e12516. https://doi. 
org/10.1111/ece.12516. 

Cohen R, Bavishi C, Haider S, Thankachen J, Rozanski A. Meta-analysis of relation of vital 
exhaustion to cardiovascular disease events. Am J Cardiol. 2017, Apr;119(8):121 1-6. https:// 
doi.org/10.1016/j.amjcard.2017.01.009. 


4 Quantifying Energy and Fatigue: Classification and Assessment of Energy and... 111 


88. 


89. 


90. 


91. 


92. 


93. 


94. 


95. 


96. 


97. 


98. 


99. 


100. 


101 


102. 


103. 


104. 


105. 


106. 


107. 


Nagy A, et al. Association of Exercise Capacity with physical functionality and various 
aspects of fatigue in patients with coronary artery disease. Behav Med. 2018, Jan;44(1):28—35. 
https://doi.org/10.1080/08964289.2016.1189395. 

Lewko A, Bidgood P, Jewell A, Garrod R. A comprehensive literature review of 
COPD-related fatigue. Curr Respir Med Rev. 2012, Nov;8(5):370-82. https://doi. 
org/10.2174/1573398 12803832476. 

Jensen Ø, Bernklev T, Jelsness-Jgrgensen L-P. Fatigue in type 1 diabetes: a systematic 
review of observational studies. Diabetes Res Clin Pract. 2017, Jan;123:63—74. https://doi. 
org/10.1016/j.diabres.2016.11.002. 

Vincent A, Benzo RP, Whipple MO, McAllister SJ, Erwin PJ, Saligan LN. Beyond pain in 
fibromyalgia: insights into the symptom of fatigue. Arthritis Res Ther. 2013, Nov;15(6):221. 
https://doi.org/10.1186/ar4395. 

Czuber-Dochan W, Ream E, Norton C. Review article: description and management of 
fatigue in inflammatory bowel disease. Aliment Pharmacol Ther. 2013;37(5):505—16. https:// 
doi.org/10.1111/apt.12205. 

Han CJ, Yang GS. Fatigue in irritable bowel syndrome: a systematic review and meta-analysis 
of pooled frequency and severity of fatigue. Asian Nurs Res. 2016, Mar;10(1):1—10. https:// 
doi.org/10.1016/j.anr.2016.01.003. 

Nocerino A, Nguyen A, Agrawal M, Mone A, Lakhani K, Swaminath A. Fatigue in inflamma- 
tory bowel diseases: etiologies and management. Adv Ther. 2020, Jan;37(1):97-112. https:// 
doi.org/10.1007/s12325-019-01151-w. 

Moldovan I, et al. Pain and depression predict self-reported fatigue/energy in lupus. Lupus. 
2013, Jun;22(7):684—9. https://doi.org/10.1177/0961203313486948. 

Ghanean H, Ceniti AK, Kennedy SH. Fatigue in patients with major depressive disorder: 
prevalence, burden and pharmacological approaches to management. CNS Drugs. 2018, 
Jan;32(1):65—74. https://doi.org/10.1007/s40263-018-0490-z. 

Charvet L, Serafin D, Krupp LB. Fatigue in multiple sclerosis. Fatigue Biomed Health Behav. 
2014, Jan;2(1):3-13. https://doi.org/10.1080/21641846.2013.843812. 

Latimer-Cheung AE, et al. Effects of exercise training on fitness, mobility, fatigue, and 
health-related quality of life among adults with multiple sclerosis: a systematic review to 
inform guideline development. Arch Phys Med Rehabil. 2013, Sep;94(9):1800-1828.e3. 
https://doi.org/10.1016/j.apmr.2013.04.020. 

Newland P, Starkweather A, Sorenson M. Central fatigue in multiple sclerosis: a review of the 
literature. J Spinal Cord Med. 2016, Jul;39(4):386-99. https://doi.org/10.1080/10790268.201 
6.1168587. 

Skoie IM, et al. Fatigue in psoriasis: a controlled study. Br J Dermatol. 2017;177(2):505-12. 


. Weinstein AA, et al. Effect of aerobic exercise training on fatigue and physical activity in 


patients with pulmonary arterial hypertension. Respir Med. 2013, May;107(5):778-84. 
https://doi.org/10.1016/j.rmed.2013.02.006. 

Artom M, Moss-Morris R, Caskey F, Chilcot J. Fatigue in advanced kidney disease. Kidney 
Int. 2014, Sep;86(3):497—505. https://doi.org/10.1038/ki.2014.86. 

Stebbings S, Treharne GJ. Fatigue in rheumatic disease: an overview. Int J Clin Rheumatol. 
2010, Aug;5(4):487—502. https://doi.org/10.2217/ijr.10.30. 

do Espírito Santo RC, et al. Neuromuscular fatigue is weakly associated with percep- 
tion of fatigue and function in patients with rheumatoid arthritis. Rheumatol Int. 2018, 
Mar;38(3):415—23. https://doi.org/10.1007/s00296-017-3894-z. 

Bailes S, et al. Fatigue: the forgotten symptom of sleep apnea. J Psychosom Res. 2011, 
Apr;70(4):346—54. https://doi.org/10.1016/j.jpsychores.2010.09.009. 

Aarnes R, Stubberud J, Lerdal A. A literature review of factors associated with fatigue after 
stroke and a proposal for a framework for clinical utility. Neuropsychol Rehabil. 2019, 
Mar;0(0):1—28. https://doi.org/10.1080/09602011.2019.1589530. 

Duncan F, Wu S, Mead GE. Frequency and natural history of fatigue after stroke: a sys- 
tematic review of longitudinal studies. J Psychosom Res. 2012, Jul;73(1):18—27. https://doi. 
org/10.1016/j.jpsychores.2012.04.001. 


112 


108. 


109. 


110. 


111. 


112. 


113. 


114. 


115. 


116. 


117. 


118. 


119. 


120. 


121. 


122. 


123. 


124. 


125. 


126. 
127. 


128. 


N. L. Solomon and V. Manea 


Newland P, Oliver B, Newland JM, Thomas FP. Testing feasibility of a Mobile application to 
monitor fatigue in people with multiple sclerosis. J Neurosci Nurs. 2019, Dec;51(6):331-4. 
https://doi.org/10.1097/JNN.0000000000000479. 

Maguire R, McCann L, Miller M, Kearney N. Nurse's perceptions and experiences of using 
of a mobile-phone-based advanced symptom management system (ASyMSO) to monitor and 
manage chemotherapy-related toxicity. Eur J Oncol Nurs. 2008, Sep;12(4):380-6. https://doi. 
org/10.1016/j.ejon.2008.04.007. 

Nicholas J, Larsen ME, Proudfoot J, Christensen H. Mobile apps for bipolar disorder: a sys- 
tematic review of features and content quality. J Med Internet Res. 2015;17(8):e198. https:// 
doi.org/10.2196/jmir.4581. 

Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin 
Psychol. 2008;4(1):1—32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415. 

Larson R, Csikszentmihalyi M. The experience sampling method. In: Flow and the founda- 
tions of positive psychology. Dordrecht: Springer Netherlands; 2014. p. 21—34. 

van Berkel N, Ferreira D, Kostakos V. The experience sampling method on Mobile devices. 
ACM Comput Surv. 2018, Jan;50(6):1—40. https://doi.org/10.1145/3123988. 

Guérin E, Fortier MS, Williams T. ‘I just NEED to move...’: examining women's passion for 
physical activity and its relationship with daily affect and vitality. Psychol Well- Theory Res 
Pract. 2013, Nov;3(1):4. https://doi.org/10.1186/2211-1522-3-4. 

Zacher H, Brailsford HA, Parker SL. Micro-breaks matter: a diary study on the effects 
of energy management strategies on occupational Well-being. J Vocat Behav. 2014, 
Dec;85(3):287—97. https://doi.org/10.1016/j.jvb.2014.08.005. 

Rhee H, Kim S. Effects of breaks on regaining vitality at work: An empirical comparison of 
‘conventional’ and ‘smart phone’ breaks. Comput Hum Behav. 2016, Apr;57:160—7. https:// 
doi.org/10.1016/j.chb.2015.11.056. 

Murphy SL, Smith DM. Ecological measurement of fatigue and fatigability in older adults 
with osteoarthritis. J Gerontol Ser A. 2010, Feb;65A(2):184—9. https://doi.org/10.1093/ 
gerona/glp137. 

Hacker ED, Ferrans CE. Ecological momentary assessment of fatigue in patients receiv- 
ing intensive cancer therapy. J Pain Symptom Manag. 2007, Mar;33(3):267—75. https://doi. 
org/10.1016/j.jpainsymman.2006.08.007. 

Bower JE. Cancer-related fatigue: mechanisms, risk factors, and treatments. Nat Rev Clin 
Oncol. 2014, Oct;11(10):597—600. https://doi.org/10.1038/nrclinonc.2014.127. 

van der Linden D, Frese M, Meijman TF. Mental fatigue and the control of cognitive pro- 
cesses: effects on perseveration and planning. Acta Psychol. 2003, May;113(1):45-65. 
https://doi.org/10.1016/80001-6918(02)00150-6. 

Ackerman PL. 100 years without resting. In: Cognitive fatigue: multidisciplinary perspectives 
on current research and future applications. Washington, DC, US: American Psychological 
Association; 2011. p. 11-43. 

Lieberman HR. Cognitive methods for assessing mental energy. Nutr Neurosci. 2007, 
Oct;10(5-6):229—42. https://doi.org/10.1080/10284150701722273. 

Basner M, Dinges DF. Maximizing sensitivity of the psychomotor vigilance test (PVT) to 
sleep loss. Sleep. 2011, May;34(5):581-91. https://doi.org/10.1093/sleep/34.5.581. 
Holsbeeke L, Ketelaar M, Schoemaker MM, Gorter JW. Capacity, capability, and performance: 
different constructs or three of a kind? Arch Phys Med Rehabil. 2009, May;90(5):849—55. 
https://doi.org/10.1016/j.apmr.2008.11.015. 

World Health Organization, “World report on ageing and health.,” World Health Organ., 2015. 
Enright PL. The six-minute walk test. Respir Care. 2003;48(8):3. 

Rudick R, et al. Recommendations from the national multiple sclerosis society clinical 
outcomes assessment task force. Ann Neurol. 1997;42(3):379-82. https://doi.org/10.1002/ 
ana.410420318. 

Podsiadlo D, Richardson S. The timed ‘up & go’: a test of basic functional mobility for 
frail elderly persons. J Am Geriatr Soc. 1991;39(2):142-8. https://doi.org/10.1111/j.1532- 
5415.1991.tb01616.x. 


4 Quantifying Energy and Fatigue: Classification and Assessment of Energy and... 113 


129. 


130. 


131. 


132. 


133. 


134. 


135. 


136. 


137. 


138. 


139. 


140. 


141 


143. 


144. 


145. 


146. 


147. 


148. 


149. 


Boolani A, O’Connor PJ, Reid J, Ma S, Mondal S. Predictors of feelings of energy differ from 
predictors of fatigue. Fatigue Biomed. Health Behav. 2019, Jan;7(1):12—28. https://doi.org/1 
0.1080/21641846.2018.1558733. 

Roldán Jiménez C, Bennett P, Ortiz García A, Cuesta Vargas AI. Fatigue detection during 
sit-to-stand test based on surface electromyography and acceleration: a case study. Sensors. 
2019, Jan;19(19):19. https://doi.org/10.3390/s19194202. 

Muñoz JE, Gouveia ER, Cameiráo MS, Badia SB. PhysioLab-a multivariate physiological 
computing toolbox for ECG, EMG and EDA signals: a case of study of cardiorespiratory fit- 
ness assessment in the elderly population. Multimed Tools Appl. 2018, May;77(9):11521—46. 
https://doi.org/10.1007/s11042-017-5069-z. 

Nicklas BJ, et al. Effects of caloric restriction on cardiorespiratory fitness, fatigue, and dis- 
ability responses to aerobic exercise in older adults with obesity: a randomized controlled 
trial. J Gerontol A Biol Sci Med Sci. 2019, Jun;74(7):1084—90. https://doi.org/10.1093/ 
gerona/gly159. 

Sehle A, Vieten M, Mündermann A, Dettmers C. Difference in motor fatigue between patients 
with stroke and patients with multiple sclerosis: a pilot study. Front Neurol. 201455 https:// 
doi.org/10.3389/fneur.2014.00279. 

Dedering A, Gnospelius A, Elfving B. Reliability of measurements of endurance time, elec- 
tromyographic fatigue and recovery, and associations to activity limitations, in patients with 
lumbar disc herniation. Physiother Res Int. 2010;15(4):189—98. https://doi.org/10.1002/ 
pri.457. 

Schmitt J, Lindner N, Reuss-Borst M, Holmberg H-C, Sperlich B. A 3-week multimodal 
intervention involving high-intensity interval training in female cancer survivors: a random- 
ized controlled trial. Physiol Rep. 2016;4(3):e12693. https://doi.org/10.14814/phy2.12693. 
Folstein MF, Robins LN, Helzer JE. The mini-mental state examination. Arch Gen Psychiatry. 
1983, Jul;40(7):812. https://doi.org/10.1001/archpsyc.1983.01790060110016. 

U.S. Army, “Army individual test battery,’ Man. Dir. Scoring, 1944. 

Hannay HJ, Levin HS. Selective reminding test: An examination of the equiva- 
lence of four forms. J Clin Exp Neuropsychol. 1985, Jun;7(3):251-63. https://doi. 
org/10.1080/01688638508401258. 

Barbizet J, Cany E. Clinical and psychometrical study of a patient with memory disturbances. 
Int J Neurol. 1967;7(1):44—54. 

Rao SM, Hammeke TA, McQuillen MP, Khatri BO, Lloyd D. Memory disturbance in chronic 
progressive multiple sclerosis. Arch Neurol. 1984, Jun;41(6):625—31. https://doi.org/10.1001/ 
archneur. 1984.04210080033010. 


. A. Smith, Symbol digit modalities test. Western Psychological Services Los Angeles, 1973. 
142. 


Gronwall DMA. Paced auditory serial-addition task: a measure of recovery from concussion. 
Percept Mot Skills. 1977, Apr;44(2):367—73. https://doi.org/10.2466/pms. 1977.44.2.367. 
Barr A, Brandt J. Word-list generation deficits in dementia. J Clin Exp Neuropsychol. 1996, 
Dec;18(6):810—22. https://doi.org/10.1080/01688639608408304. 

Obrist PA, Webb RA, Sutterer JR. Heart rate and somatic changes during aversive condition- 
ing and a simple reaction time task. Psychophysiology. 1969;5(6):696—723. 

Bever CT Jr, Grattan L, Panitch HS, Johnson KP. The brief repeatable battery of neuropsycho- 
logical tests for multiple sclerosis: a preliminary serial study. Mult Scler J. 1995;1(3):165—9. 
Smolders KC, de Kort YA. Bright light and mental fatigue: effects on alertness, vitality, per- 
formance and physiological arousal. J Environ Psychol. 2014;39:77-91. 

Westbrook JI, Raban MZ, Walter SR, Douglas H. Task errors by emergency physicians are 
associated with interruptions, multitasking, fatigue and working memory capacity: a prospec- 
tive, direct observation study. BMJ Qual Saf. 2018;27(8):655—63. 

Hartzler BM. Fatigue on the flight deck: the consequences of sleep loss and the benefits of 
napping. Accid Anal Prev. 2014;62:309-18. 

Staniute M, Bunevicius A, Brozaitiene J, Bunevicius R. Relationship of health-related qual- 
ity of life with fatigue and exercise capacity in patients with coronary artery disease. Eur J 
Cardiovasc Nurs. 2014, Aug;13(4):338-44. https://doi.org/10.1177/14745 15113496942. 


114 


150. 


151. 


152. 


153. 


154. 


155. 


156. 


157. 


158. 


159. 


160. 


161. 


162. 


163. 


164. 


165. 


166. 


167. 


168. 


169. 
170. 


171. 


N. L. Solomon and V. Manea 


Pimenta A, Carneiro D, Novais P, Neves J. Monitoring mental fatigue through the analysis of 
keyboard and mouse interaction patterns. In: International Conference on Hybrid Artificial 
Intelligence Systems; 2013. p. 222-31. 

Foti K, Xanthopoulou D, Papagiannidis S, Kafetsios K. The role of tweet-related emotion on 
the exhaustion-recovery from work relationship. In: Conference on e-Business, e-Services 
and e-Society; 2019. p. 380-91. 

Dikmen S, Machamer J, Temkin N, McLean A. Neuropsychological recovery in patients 
with moderate to severe head injury: 2 year follow-up. J Clin Exp Neuropsychol. 
1990;12(4):507-19. 

Ponsford JL, Olver JH, Curran C. A profile of outcome: 2 years after traumatic brain injury. 
Brain Inj. 1995;9(1):1—10. 

Van Zomeren AH, Van den Burg W. Residual complaints of patients two years after severe 
head injury. J Neurol Neurosurg Psychiatry. 1985;48(1):21-8. 

Price E, Moore G, Galway L, Linden M. Towards a mobile assistive technology for monitor- 
ing and assessing cognitive fatigue in individuals with acquired brain injury. In: 2015 IEEE 
International Conference on Computer and Information Technology; Ubiquitous Computing 
and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence 
and Computing; 2015. p. 1487-91. 

Tran Y, Wijesuriya N, Tarvainen M, Karjalainen P, Craig A. The relationship between spectral 
changes in heart rate variability and fatigue. J Psychophysiol. 2009;23(3):143-51. 
Segerstrom SC, Nes LS. Heart rate variability reflects self-regulatory strength, effort, and 
fatigue. Psychol Sci. 2007;18(3):275-81. 

Blausen.com staff. Medical gallery of Blausen medical 2014. WikiJournal of Medicine. 
2014;1(2) 

Samel A, Wegmann H-M, Vejvoda M. Aircrew fatigue in long-haul operations. Accid Anal 
Prev. 1997;29(4):439—52. 

Goldman LI, McDonough MT, Rosemond GP. Stresses affecting surgical performance 
and learning: I. correlation of heart rate, electrocardiogram, and operation simultaneously 
recorded on videotapes. J Surg Res. 1972;12(2):83-6. 

Chen C, Li K, Wu Q, Wang H, Qian Z, Sudlow G. EEG-based detection and evaluation of 
fatigue caused by watching 3DTV. Displays. 2013;34(2):81-8. 

Holdsworth L, Evens T. Is heart rate variability a useful marker of stress and fatigue in emer- 
gency and pre-hospital clinicians?—a systematic review. Resuscitation. 2017;118:e95. 
Stamler JS, Goldman ME, Gomes J, Matza D, Horowitz SF. The effect of stress and fatigue 
on cardiac rhythm in medical interns. J Electrocardiol. 1992;25(4):333-8. 

Heemskerk J, et al. Relax, it’s just laparoscopy! A prospective randomized trial on heart rate 
variability of the surgeon in robot-assisted versus conventional laparoscopic cholecystec- 
tomy. Dig Surg. 2014;31(3):225-32. 

Jung S-J, Shin H-S, Chung W-Y. Driver fatigue and drowsiness monitoring system with embed- 
ded electrocardiogram sensor on steering wheel. IET Intell Transp Syst. 2014;8(1):43—50. 
Díaz-Toral LG, Banderas-Dorantes TR, Rivas-Vilchis JF. Impact of electroacupuncture 
treatment on quality of life and heart rate variability in fibromyalgia patients. J Evid-Based 
Complement Altern Med. 2017;22(2):216-22. 

DeLuca J. “19 Fatigue: its definition, its study, and its future,’ Fatigue Window Brain, 
p. 319, 2005. 

Stern JA, Boyer D, Schroeder D. Blink rate: a possible measure of fatigue. Hum Factors. 
1994;36(2):285-97. 

Sirois S, Brisson J. Pupillometry. Wiley Interdiscip Rev Cogn Sci. 2014;5(6):679—92. 
Eggert T, et al. The pupillographic sleepiness test in adults: effect of age, gender, and time of 
day on pupillometric variables. Am J Hum Biol. 2012;24(6):820-8. 

Lowenstein O, Feinberg R, Loewenfeld IE. Pupillary movements during acute and chronic 
fatigue: a new test for the objective evaluation of tiredness. Invest Ophthalmol Vis Sci. 
1963;2(2):138—57. 


4 Quantifying Energy and Fatigue: Classification and Assessment of Energy and... 115 


172. 


173. 


174. 


175. 


176. 


177. 


178. 


179. 


180. 


181. 


182. 


183. 


184. 


185. 


186. 


187. 


188. 


189. 


190. 


191. 
192. 


193. 


Hirvonen K, Puttonen S, Gould K, Korpela J, Koefoed VF, Miiller K. Improving the saccade 
peak velocity measurement for detecting fatigue. J Neurosci Methods. 2010;187(2): 199-206. 
You C-W, et al. CarSafe: a driver safety app that detects dangerous driving behavior using 
dual-cameras on smartphones. In: Proceedings of the 2012 ACM Conference on Ubiquitous 
Computing; 2012. p. 671-2. 

Zhang Y-F, Gao X-Y, Zhu J-Y, Zheng W-L, Lu B-L. A novel approach to driving fatigue 
detection using forehead EOG. In: 2015 7th International IEEE/EMBS Conference on Neural 
Engineering (NER); 2015. p. 707-10. 

Morris G, Maes M. Myalgic encephalomyelitis/chronic fatigue syndrome and encephalomy- 
elitis disseminata/multiple sclerosis show remarkable levels of similarity in phenomenology 
and neuroimmune characteristics. BMC Med. 2013;11(1):1—23. 

Lorist MM, Bezdan E, ten Caat M, Span MM, Roerdink JB, Maurits NM. The influence of 
mental fatigue and motivation on neural network dynamics; an EEG coherence study. Brain 
Res. 2009;:1270:95-106. 

Sheng Y, Huang K, Zou J, Wang L, Wei P. Exploring the relationship between neural mecha- 
nism and detection in mental fatigue by genetic algorithm and hierarchical clustering. In: 
2019 IEEE 7th International Conference on Bioinformatics and Computational Biology 
(ICBCB); 2019. p. 120-4. 

Craig A, Tran Y, Wijesuriya N, Nguyen H. Regional brain wave activity changes associated 
with fatigue. Psychophysiology. 2012;49(4):574—82. 

Zhao C, Zheng C, Zhao M, Tu Y, Liu J. Multivariate autoregressive models and kernel learn- 
ing algorithms for classifying driving mental fatigue based on electroencephalographic. 
Expert Syst Appl. 2011;38(3):1859-65. 

Kahol K, Smith M, Brandenberger J, Ashby A, Ferrara JJ. Impact of fatigue on neurophysi- 
ologic measures of surgical residents. J Am Coll Surg. 2011;213(1):29-34. 

Guru KA, Shafiei SB, Khan A, Hussein AA, Sharif M, Esfahani ET. Understanding cognitive 
performance during robot-assisted surgery. Urology. 2015;86(4):751-7. 

Gold AE, MacLeod KM, Deary IJ, Frier BM. Hypoglycemia-induced cognitive dysfunction 
in diabetes mellitus: effect of hypoglycemia unawareness. Physiol Behav. 1995;58(3):501-11. 
Newsholme EA, Blomstrand E, Ekblom B. Physical and mental fatigue: metabolic mecha- 
nisms and importance of plasma amino acids. Br Med Bull. 1992;48(3):477-95. 

Olsson EM, Roth WT, Melin L. Psychophysiological characteristics of women suffering 
from stress-related fatigue. Stress Health J Int Soc Investig Stress. 2010;26(2):113-26. 
Kuebler U, et al. Norepinephrine infusion with and without alpha-adrenergic blockade by 
phentolamine increases salivary alpha amylase in healthy men. Psychoneuroendocrinology. 
2014;49:290-8. 

Hopstaken JF, van der Linden D, Bakker AB, Kompier MA. The window of my eyes: task dis- 
engagement and mental fatigue covary with pupil dynamics. Biol Psychol. 2015;110:100—6. 
Van Heukelom RO, Prins JB, Smits MG, Bleijenberg G. Influence of melatonin on fatigue 
severity in patients with chronic fatigue syndrome and late melatonin secretion. Eur J Neurol. 
2006;13(1):55—60. 

Merikangas KR, et al. Real-time mobile monitoring of the dynamic associations among 
motor activity, energy, mood, and sleep in adults with bipolar disorder. JAMA Psychiatry. 
2019;76(2):190-8. 

Bloch KE. Polysomnography: a systematic review. Technol Health Care. 1997;5(4):285—305. 
Sadeh A, Hauri PJ, Kripke DF, Lavie P. The role of actigraphy in the evaluation of sleep 
disorders. Sleep. 1995;18(4):288—302. 

Wac K. From quantified self to quality of life. In: Digital health. Springer; 2018. p. 83-108. 
Rizzo D, et al. The role of fatigue and sleepiness in drivers with obstructive sleep apnea. 
Transp Res Part F Traffic Psychol Behav. 2019;62:796—804. 

Fernandes-Junior SA, Ruiz FS, Antonietti LS, Tufik S, Tálio de Mello M. Sleep, fatigue and 
quality of life: a comparative analysis among night shift workers with and without children. 
PLoS One. 2016;11(7):e0158580. 


116 


194. 


195. 


196. 


197. 


198. 


199. 


200. 


201. 


202. 


203. 


204. 


205. 


206. 


207. 


208. 


209. 


210. 


211. 


212. 


213. 


214. 


N. L. Solomon and V. Manea 


Campbell R, Tobback E, Delesie L, Vogelaers D, Mariman A, Vansteenkiste M. Basic psy- 
chological need experiences, fatigue, and sleep in individuals with unexplained chronic 
fatigue. Stress Health. 2017;33(5):645—55. 

Maher C, et al. Fatigue is a major issue for children and adolescents with physical disabilities. 
Dev Med Child Neurol. 2015;57(8):742-7. 

Attarian HP, Brown KM, Duntley SP, Carter JD, Cross AH. The relationship of sleep distur- 
bances and fatigue in multiple sclerosis. Arch Neurol. 2004, Apr;61(4):525—8. https://doi. 
org/10.1001/archneur.61.4.525. 

Kaynak H, et al. Fatigue and sleep disturbance in multiple sclerosis. Eur J Neurol. 
2006;13(12):1333-9. 

Veauthier C, et al. Fatigue in multiple sclerosis is closely related to sleep disorders: a poly- 
somnographic cross-sectional study. Mult Scler J. 2011;17(5):613-22. 

Kaminska M, et al. Obstructive sleep apnea is associated with fatigue in multiple sclerosis. 
Mult Scler J. 2012;18(8):1159—69. 

Shitrit AB-G, et al. Sleep disturbances can be prospectively observed in patients with an inac- 
tive inflammatory bowel disease. Dig Dis Sci. 2018;63(11):2992-7. 

Sun J-L, Lin C-C. Relationships among daytime napping and fatigue, sleep quality, and qual- 
ity of life in cancer patients. Cancer Nurs. 2016;39(5):383-92. 

Ancoli-Israel S, et al. Sleep, fatigue, depression, and circadian activity rhythms in women 
with breast cancer before and after treatment: a l-year longitudinal study. Support Care 
Cancer. 2014, Sep;22(9):2535-45. https://doi.org/10.1007/s00520-014-2204-5. 

Holliday EB, Dieckmann NF, McDonald TL, Hung AY, Thomas CR Jr, Wood LJ. Relationship 
between fatigue, sleep quality and inflammatory cytokines during external beam radiation 
therapy for prostate cancer: a prospective study. Radiother Oncol. 2016;:118(1):105-11. 
Cambras T, Castro-Marrero J, Zaragoza MC, Díez-Noguera A, Alegre J. Circadian rhythm 
abnormalities and autonomic dysfunction in patients with chronic fatigue syndrome/Myalgic 
encephalomyelitis. PLoS One. 2018;13(6):e0198106. 

Nicklas BJ, Beavers DP, Mihalko SL, Miller GD, Loeser RF, Messier SP. Relationship 
of objectively-measured habitual physical activity to chronic inflammation and fatigue in 
middle-aged and older adults. J Gerontol Ser Biomed Sci Med Sci. 2016;71(11):1437-43. 
Nilsson M, Arving C, Thormodsen I, Assmus J, Berntsen S, Nordin K. Moderate-to-vigorous 
intensity physical activity is associated with modified fatigue during and after cancer treat- 
ment. Support Care Cancer. 2019:1-8. 

Vancampfort D, et al. Higher cardio-respiratory fitness is associated with increased mental 
and physical quality of life in people with bipolar disorder: a controlled pilot study. Psychiatry 
Res. 2017;256:219-24. 

Sheshadri A, Kittiskulnam P, Johansen KL. Higher physical activity is associated with less 
fatigue and insomnia among patients on hemodialysis. Kidney Int Rep. 2019;4(2):285-92. 
Qazi T, Farraye FA. Sleep and inflammatory bowel disease: an important bi-directional rela- 
tionship. Inflamm Bowel Dis. 2019;25(5):843-52. 

Sofia MA, Andersen M, Rubin DT. Sleep dysfunction and inflammatory bowel disease. In: 
Sleep effect on gastrointestinal health and disease: translational opportunities for promoting 
health and optimizing disease management. Nova Science Publishers, Inc; 2018. p. 177-99. 
Abbott L. Energy through motion\copyright: An activity intervention for cancer-related 
fatigue in an ambulatory infusion center. Clin J Oncol Nurs. 2017;21(5):618. 

Ren D, An Y, Li Z. The discriminative model of mental fatigue based on comprehensive 
parameter analysis. In: 2019 Prognostics and System Health Management Conference 
(PHM-Qingdao); 2019. p. 1-6. 

Smith MR, Chai R, Nguyen HT, Marcora SM, Coutts AJ. Comparing the effects of three 
cognitive tasks on indicators of mental fatigue. J Psychol. 2019;153(8):759-83. 

Brown DM, Bray SR. Effects of mental fatigue on exercise intentions and behavior. Ann 
Behav Med. 2019;53(5):405-14. 


4 Quantifying Energy and Fatigue: Classification and Assessment of Energy and... 117 


215. 


216. 


217. 


218. 


219. 


220. 


221. 


222. 


223. 
224. 


225. 
226. 


221. 


228. 


229. 


230. 


231. 


Kanitz JL, et al. The impact of eurythmy therapy on fatigue in healthy adults-a controlled 
trial. Eur J Integr Med. 2012;4(3):e289-97. 

Smolders KC, De Kort YA, Cluitmans PJM. A higher illuminance induces alertness even dur- 
ing office hours: findings on subjective measures, task performance and heart rate measures. 
Physiol Behav. 2012;107(1):7-16. 

Oriyama S, Miyakoshi Y, Kobayashi T. Effects of two 15-min naps on the subjective sleepi- 
ness, fatigue and heart rate variability of night shift nurses. Ind Health. 2013; 

Singh H, et al. Robotic surgery improves technical performance and enhances prefrontal 
activation during high temporal demand. Ann Biomed Eng. 2018;46(10):1621—36. 

Dishman RK, Thom NJ, Puetz TW, O'Connor PJ, Clementz BA. Effects of cycling exercise 
on vigor, fatigue, and electroencephalographic activity among young adults who report per- 
sistent fatigue. Psychophysiology. 2010;47(6):1066-74. 

Zeng Z, Pantic M, Roisman GI, Huang TS. A survey of affect recognition methods: audio, 
visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell. 2008;31(1):39—58. 
Braley TJ, Chervin RD. Fatigue in multiple sclerosis: mechanisms, evaluation, and treatment. 
Sleep. 2010;33(8):1061-7. 

Adinolfi A. Assessment and treatment of HIV-related fatigue. J Assoc Nurses AIDS Care. 
2001, Sep;12:29—34. https://doi.org/10.1016/S 1055-3290(06)60155-6. 

Wac K. Quality of life technologies. NY USA: Encycl. Behav. Med. Springer N. Y; 2020. 
Swan M. The quantified self: fundamental disruption in big data science and biological dis- 
covery. Big Data. 2013;1(2):85-99. 

Neuringer A. Self-experimentation: a call for change. Behaviorism. 1981;9(1):79-94. 

Jones S. “Health & Fitness Wearables,” 2014. https://www.juniperresearch.com/research- 
store/devices-technology/digital-health-wearables-research-report. Accessed Jun. 19, 2020. 
Piwek L, Ellis DA, Andrews S, Joinson A. The rise of consumer health wearables: promises 
and barriers. PLoS Med. 2016;13(2):e1001953. 

Vogels EA. About one-in-five Americans use a smart watch or fitness tracker. Wash. DC Pew 
Res. Cent; 2019. 

Ubrani J, Llamas R, Shirer M. “IDC forecasts sustained double-digit growth for wearable 
devices led by steady adoption of smart-watches,” International Data Corporation, 2020. 
https://www.idc.com/getdoc.jsp?containerId-p. Accessed Jun. 15, 2020. 

Estrada-Galifianes V, Wac K. “Collecting, exploring and sharing personal data: why, how and 
where,” Data Sci., no. Preprint, pp. 1-28. 

Stansberry K, Anderson J, Rainie L. Experts optimistic about the next 50 years of digital 
life. Pew Research Center: Internet, Science & Tech; 2019. https://www.pewresearch.org/ 
internet/2019/10/28/experts-optimistic-about-the-next-50-years-of-digital-life/ ^ (accessed 
Jun. 25, 2020) 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter's Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter's Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


Chapter 5 A 
Quantifying Mobility in Quality of Life inim 


Nancy E. Mayo and Kedar K. V. Mate 


What Is Mobility? 


According to Merriam-Webster's Dictionary mobility is the "ability or capacity 
to move".! 

In the scientific community mobility has been defined “as the ability to move 
oneself (either independently or by using assistive devices or transportation) within 
environments that expand from one's home to the neighborhood and to regions 
beyond" [1]. The life-space within a person can move has also been recognized as 
ranging from the person's room, home, outdoors, and neighborhood to the service 
community of shops, banks, healthcare facilities etc., surrounding area within per- 
son's own country, and the world. This mobility is recognized to be constrained or 
influenced by financial, environmental, and psychosocial conditions as well as 
physical and cognitive capabilities. Gender, culture, and the person's life-experience 
also affect mobility. In order to conceptualize mobility more coherently, Webber 
et al. [1] proposed a framework that links factors relevant to walking, wheeling, 
driving, and taking alternate forms of transportation within different life-spaces. 

In the context of global and public health, mobility has been defined within the 
World Health Organization's International Classification of Function, Disability 
and Health (ICF) [2] according to the components of: changing and maintaining 
body position (d410-d429); carrying, moving and handling objects (d430-d449); 
walking and moving (d450-d469);moving around using transportation (d470-d489). 
Table 5.1 lists these components that are based on the person doing the movement. 


! https://www.merriam-webster.com/dictionary/mobility 
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Table 5.1 Components of 


xS 1 Mobility component [ICF code] 
Mobility according to the ICF i 


Changing and maintaining body position 


_[d410] changing basic body position 


_[d415 maintaining a body position 
f [d420] transferring oneself 
Carrying, moving and handling objects 


_[d430] lifting and carrying objects 


[d435 moving objects with lower extremities 
[440] Fine hand use 
[d445 hand and arm use 
Walking and moving 
-[d450] walking 
[d455] moving around 


l [d460] moving around in different locations 


_[d465] moving around using equipment 


What is immediately obvious from the definition using the ICF is, that mobility is a 
necessary (but not sufficient) capacity required for many other activities such as 
basic activities of daily living, more complex activities required for maintain self 
and living space, work, and recreation and leisure including sports. These down- 
stream activities that depend on some degree of mobility are themselves important 
contributor to QOL [3]. 

The ICF definition of mobility also includes two other very important qualifying 
constructs: capacity, what the person can do usually in a test situation; and perfor- 
mance, what the person actually does [4]. Capacity and performance constructs 
have important implications for measuring mobility. In the ICF, capacity refers to 
the person’s ability to execute a task in a standard environment. This tends to refer 
to clinical testing. Whereas Webber et al. [1] also include capacity as a factor in their 
mobility model but refers to having the biological capability such as having suffi- 
cient joint mobility or strength to make mobility possible, areas that fall under the 
body structure and function component of the ICF. 

In the ICF context, performance is poorly described. In another view, Loechte 
et al. [5] considered mobility in relationship to movement away from the home and 
other parameters related to what the person is doing. Action range is quantified as to 
how far a person moves outside their house. Distance is a parameter that can be 
measured independently from the home reference point such as distance covered 
per unit time such as per minutes (Six Minute Walk Test (6MWT) [6], per hour, per 
day. Other mobility parameters are duration (how long someone is mobile), pace 
(how many steps per minute, e.g. 100 steps per minute), and frequency of mobility 
events (how many 10 minute bouts of walking). There are important mobility 
parameters that are related to physical activity guidelines [7]. Mobility can also be 
characterized qualitatively, without measurement units such as time-of-day, alone 
or accompanied, and places. For example, places people move to more than 5 min- 
utes can be captured and qualified by location (parks, malls, cafes, museums), 
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activity (socializing, exercising), environment (noisy), or either usual or novel loca- 
tion. As an example, consider two different people. 

Mark leaves the house at the same time every day, rides his bicycle 4 km to work 
taking the same route, and comes home. On the weekend he goes to the gym and 
does family-related errands and activities. 

Eleanor, works from home and regularly exercises by walking, biking, or swim- 
ming at her community pool; she sometimes calls up a friend to go with her. These 
activities are done at different times during the day depending on her work sched- 
ule. Sometimes she works in cafes and makes a point of going to different ones. On 
the weekend, she likes to explore different parts of the city or take small trips out of 
the city. These are done alone or sometimes with a friend. 

In each case, Mark and Eleanor would be classified with the same mobility 
capacity and performance indicators on the ICF, and they would have the same 
“action distance” values, but their mobility is realized through very different pat- 
terns. The richness of this mobility variety is not easy to capture without technology. 
Mark may accurately self-report because of the routine nature of his mobility but 
Eleanor would not be able to provide an average mobility rating for the past week 
or past month owing to her mobility variation. 

While capacity ICF indicators are necessary in the context of a person with a 
health conditions, they are not sufficient for tracking mobility in healthy popula- 
tions where performance in the real world is the relevant QOL indicator. 


What Is QOL? 


The Dictionary of Quality of Life and Health Outcomes Measurement [8] has this 
to say about QoL: 


QOL is a term often used erroneously to refer to health-related quality of life or health 
status, but is broader than just health and includes components of material comforts, health 
and personal safety, relationships, learning, creative expression, opportunity to help and 
encourage others, participation in public affairs, socializing, and leisure [9]. The World 
Health Organization (WHO) has defined quality of life as ‘individuals’ perception of their 
position in life in the context of the culture in which they live and in relation to their goals, 
expectations, standards and concerns". In the context of health research, quality of life goes 
beyond a description of health status, but rather is a reflection of the way that people per- 
ceive and react to their health status and to other, nonmedical aspects of their lives. 
According to Aristotle, quality of life would be the best kind of life, the happiest life [10]. 


Clearly measuring this happiest of lives is a challenge. QoL is not the same as health 
as health is only one of many QoL components [9]. Three approaches have been 
taken to measure QoL. QoL profiles are derived from measures that are made up of 
multi-item domains that produce domain scores and a total score. The WHOQOL-100 
[11] (100 items; six domains: physical health, psychological, level of independence, 
social relationships, environment, spirituality, plus two single indicators of general 
health and global QoL) and WHOQOL-Bref [12] (26 items; four domains: physical 
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health, psychological, social relationships, environment and the same two single 
indicators), are examples of profile QoL measures where the domains are physical 
health (including mobility), psychological health, social relationships and the envi- 
ronment. The CASP-19 is a quality-of-life measure comprising four domains, con- 
trol, autonomy, pleasure and self-realization [13]. QoL has also been measured 
using health indices such as the EQ-5D from the EUROQOL Group [14] or the 
Health Utilities Index [15] but these measures do not cover many, if any, domains 
beyond health [16]. The advantage of indices is that they comprise multiple dimen- 
sions (usually with one item per each dimension) and a single score is derived by 
weights related to how much a typical citizen is willing to trade off years of life for 
these dimensions health. The third way in which QoL is measured is by a single 
item rated on an ordinal scale (e.g. Excellent, Very good, Good, Fair, Poor) or a 
visual analogue scale (from worst, 0, to best, 10).Only the person can assess their 
QoL using these methods and, other than the single-item method, only periodic 
assessments are possible [17]. 

Instead of relying on people to periodically sample and report on their QoL, Wac 
[18] proposes a new way of measuring QOL, using technologies. Quality of Life 
Technologies (QoLT) refers any technologies that can be used for assessment or 
improvement of the individual's QoL. QoLT will be the way of the future owing to 
the increasing availability of miniaturized computing, storage, and communication 
capacity that are now embedded within various personal devices and made available 
through smartphones and wearables. 

How much can we infer about QoL by quantifying domains related to QoL? This 
chapter addresses this novel way of thinking about QoL. Here we address one of 
these quantifiable domains, namely individual's (body) mobility, a construct that 
lends itself well to being quantified by harnessing the power of these existing and 
emerging technologies. 


Is Mobility Important for QoL? 


Now that mobility has been described in terms of capacity and performance, and 
QoL has been defined as “the happiest kind of life", we discuss if mobility is impor- 
tant to QoL and, if so, what aspects of mobility are important. The answer to this 
question needs to consider who is being asked and how it is asked. For mostly 
healthy members of the general population, mobility, particularly walking, is the 
most important of five key HRQL items. A well known and widely used measure of 
HRQL is the EQ-5D from the EUROQOL [14] group. The weight, in terms of 
degree of detraction from perfect health, that members of the general population 
(from the United States) put on having no mobility as represented by being unable 
to walk about, is —0.558 (on a scale from 0 to 1; where '0' is the worst possible 
health state and ‘1’ is perfect health). In contrast, the detracting effect of extreme 
problems in self-care (—0.471), usual activities (—0.374), pain (—0.537) and mood 
(—0.450). This translates to be willing to trade off nearly 6 years of life in order to 
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live with no problems with walking about. Research on the effect of mobility limita- 
tions on risk of death shows that people, again from the general population of the 
United States, who walk very slowly have a risk of death 1.89 times higher (95% CI: 
1.46-2.46), in comparison to the fastest walkers Liu [19]. 

In the EQ-5D classification system, the walking component of mobility is one of 
five HRQL areas clearly underlining its importance. One way of knowing if mobil- 
ity is important to QoL is to identify how often and how comprehensively mobility 
is included in recognized QoL or health-related QoL measures. Mayo et al. [16] 
found that in generic QoL or HRQL measures, mobility was represented in 1/8 
items in the HUI [20]; 10/36 items in the SF-36 (1/12 in SF-12) [21, 22]; 2/35 items 
or 1/8 dimensions of the AQOL [23], 1/26 items in the WHOQOL-BREF [12], and 
9/71 items of the QWB [24]. 

Mobility is clearly important for people with health conditions. Almost every 
health condition can have an effect on mobility, permanent or transient. There are 
ICF core sets for some 30 health conditions and all except mental health conditions 
included one or more aspects of mobility.^? Mobility is more important to QoL once 
it is limited. People tend to take mobility for granted until the limitations set in, but 
when asked how they would imaging their life without mobility, they imagine it 
poorer than with other health challenges. This is one of the reasons why, when valu- 
ing health for the purposes of allocation of scarce resources or evaluation of cost- 
effectiveness of medical interventions, those with the health condition under 
consideration are not asked to provide a valuation from their point of view as this 
valuation is considered to be too influenced by their current health state [25—27]. 

The importance of mobility to QoL has been investigated extensively using many 
different methods, quantitative and qualitative, and in many different health 
conditions. 


What Do People Say? 


One of the best ways of answering the question about how mobility relates to QoL 
is to ask people directly. This is made possible through a synthesis of the qualitative 
literature or when people are asked open-ended questions. 

A synthesis of 11 qualitative studies on QoL after hip fracture [28] identified 
mobility as a key contributor. Limited mobility affected this population's opportuni- 
ties to make free choices about their activities and social interactions, impacted on 
independence, and was a threat to preservation of self-image. Thirteen qualitative 
studies from people with leg ulcers also confirmed the importance of mobility to 
QoL [29]. 


*https://www.merriam-webster.com/dictionary/mobility 
$https://www.icf-research-branch.org/icf-core-sets 
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In a systematic review of 20 papers using an individualized approach to identify- 
ing areas important for QoL in people with cancer, Aburub et al. [30] found that in 
11 or the 20 papers, mobility/physical activity was listed a one of the top 10 areas. 
In a review of the 10 most important areas of QoL across four health conditions, 
Mayo et al. [31] reported that for people with stroke, mobility was the number one 
area of importance; for people with Multiple Sclerosis mobility was number 6, and 
for people with cancer and HIV mobility was also selected as important to QoL. 


What Do the Data Say about the QoL of the Body 


The quantitative investigation of the importance of mobility to QoL is challenging 
because, as conceptualized using the ICF model [2], mobility limitations are caused 
by impairments of body structure and function and also act to limit other important 
activities and restrict participation in key personal, family, and societal roles that 
have a more direct influence on QoL. Figure 5.1 show these theoretical influences 
by combining the ICF model with the Wilson-Cleary model [32]. The ICF focuses 
on the observable manifestations of disability, while the Wilson-Cleary model goes 
beyond these to consider the effects on health perception and QOL, considered by 
the ICF model to reflect satisfaction with the observable manifestations and akin to 
well-being. 

Having established that mobility is definitely important for QOL, other questions 
arise. Is mobility more, less, or as important as other health, social, and environmen- 
tal domains? What are the best methods of partitioning out the role of mobility in 
quality of life? If we were to use mobility to quantify QOL, how much will we 
under or overestimate QOL and in whom and under what circumstances? 
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Fig. 5.1 Integration of the Wilson-Cleary Model and the ICF showing the place of mobility 
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In order to sort out the relative importance of mobility and other factors to QOL 
it is necessary to have a strong theoretical model linking both the capacity and per- 
formance aspects of mobility to QOL. One such model is the Wilson-Cleary model 
[32] that has been shown in Fig. 5.1. The Wilson-Cleary model shows the links 
between biological and physiological measures taken on the body, symptoms 
reported by the person (pain, fatigue, mood), function (what the person can do phys- 
ically and mentally), health perception (how the person actually feels) and QOL and 
also shows that these links are affected by factors related to the person (age and sex 
but also beyond these to include lifestyle and preferences), and their environment. 
This complexity requires that mobility be considered in a multi-factorial frame- 
work and Structural Equation Modeling (SEM) [33] is an ideal statistical method 
for carrying out a fair assessment of the impact of mobility. However, it is important 
when using SEM that the outcome is a QOL measure and not a composite measure 
that includes the constructs under investigation. For example, the WHOQOL-Bref 
includes a domain for physical health and this includes items about mobility. A 
selection of 18 relevant papers from a structured review are summarized in Table 5.2. 


Table 5.2 Studies Using Structural Equation Modeling to Estimate the Importance of Mobility to 
QOL and Related Constructs 


Study #/ First | | N Model | Mobility 
Author Country Population | Number | Variables Rank 

1. Li [34] China | Older persons | 4245 3 | 1 
Outcome: Life | Other model variables: Duration of disability, | Mobility variable: 
satisfaction social engagement Difficulty with running, 


walking, climbing stairs, 
bending, reaching, and 
lifting objects 


2. Shahrbanian | Canada Multiple 188 4 1 
[35] sclerosis 
Outcome: Other model variables: Fatigue, pain, mood Mobility variable: 
Participation Physical function 
3. Alonso [36] |22 countries: 5 low and | General 51,344 |9 1 
lower-middle income; | population 
5 upper-middleincome; 
11 high income. 


Outcome: Other model variables: Chronic conditions (9 | Mobility variable: WHO 
Overall mental, 10 physical), domains of WHO-DAS disability assessment 
physical and (cognition, self-care, getting along, family schedule 2.0 (WHODAS) 
mental health | burden, stigma, life activities, participation) mobility domain: Standing 
(0-100 VAS) along with age, sex, employment, country 30 minutes, rising from 


chair, moving around 
home, leaving home, 
walking long distance. 
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Table 5.2 (continued) 


Study # / First N Model Mobility 
Author Country Population | Number | Variables Rank 
4. Lampinen Finland Older adults | 663 5 1 

[37] 

Outcome: Other model variables: Physical activity, leisure | Mobility variable: 

Mental activity, age, chronic illness, Difficulty with climbing 

Well-being stairs and walking 2 km 

(revised Beck without stopping. 

depression 

inventory) 

5. Bentley [38] | USA Older persons | 677 2 PCS 2, 
MCS 1 

Outcome: Other model variables: Functional status Mobility variable: 

PCS, MCS Life-space mobility 

6. Huang [39] | Taiwan Osteoporosis | 161 10 PCS 1, 
MCS 3 

Outcome: Other model variables: Disease characteristics | Mobility variable: ADL, 

PCS, MCS (duration, pain level, chronic diseases, fracture exercise habits 

experiences, and ADL), social support 
dimensions, and PQoL and MQoL of 
osteoporosis patients with (age, marital status, 
school years, income, and exercise habits). 

7. Lee [40] Korea Parkinson's | 217 6 Tests: 6; 
motor 
signs: | 

Outcome: Other model variables: Age, disease-related Mobility variable: Grip 

PDQL factors (motor signs, disease duration), quality of | strength, balance, 


sleep, pain, and depression 


functional reach motor 
signs: Speech, facial 
expression, tremor, 
rigidity, finger tapping, 
rapid alternating 


movements 
8. Tannenbaum | Canada Older persons | 2311 5 2 
[41] 
Outcome: Life style latent (perception health living, Mobility variable: SF-12 
Latent SRH exercise, nutrition), health conditions/ (PCS) 
(SF-12, polypharmacy, social health, mental health 
EQ-VAS) (SF-12 MCS) 


9. Kalpinski 
[42] 


USA Traumatic 312 


brain injury 


Outcome: Life 
satisfaction, 
SRH 


Other model variables: Injury severity, FIM 
cognition, FIM Independence, occupational 
activity, social engagement 


Mobility variable: Ability 
to and frequency of freely 
moving around residence 
and community including 
using transportation 


10. Shim [43] 


Rheumatic 360 


disease 


Korea 


5 2 
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Table 5.2. (continued) 
Study # / First N Model Mobility 
Author Country Population | Number | Variables Rank 
Outcome: Other model variables: Pain, pain Mobility variable: Health 
WHOQOL- catastrophising, depression, fear avoidance assessment questionnaire 
BREF beliefs (HAQ): Degree of 
difficulty with dressing, 
rising, eating, walking, 
maintaining hygiene, 
reaching, gripping, and 
other common activities) 
11. Barclay Canada Stroke 227 4 2 
[44] 
Outcome: Other model variables: Indoor and outdoor Mobility variable: Gait 
SRH (EQ-VAS) | mobility, depression speed 
12. Bouchard | Canada Multiple 189 9 2,4,5,6,9 
[45] sclerosis 
Outcome: Other model variables: Fatigue, depression, Mobility variable: 
Illness pain, health perception Balance [2], GMWT, 
intrusiveness Physical function, Power 
13. Perruccio Canada Join 449 3 3 
[46] replacement 
Outcome: Other model variables: Two latents: Mental Mobility variable: 
SRH health (anxiety, depression); social health Physical health latent: 
(participation, transportation) Pain on activity, ADL, 
physically demanding 
activities, fatigue 
14. Aree-Ue Thailand Osteoarthritis | 200 4 3 
[47] 
Outcome: OA | Other model variables: Pain, fatigue, Mobility variable: 
knee and hip depression Timed-up-and-Go test 
quality of life 
(OAKHQOL) 
15. Mayo [48] | Canada Stroke 533 4 4 
Outcome: Other model variables: Latent variables for: Mobility variable: 
SRH Biological variables, symptoms, function, health | Physical function 
perception, personal and environmental factors 
following Wilson-Cleary model 
16. Soh [49] Australia Parkinson's |210 12 4,7 
Outcome: Other model variables: Age, co-morbidities, Mobility variable: Motor 
PDQ-39 disease duration, disease severity, fall history, impairments, (UPDRS), 


sex, social support, motor and non-motor 
impairment (UPDRS), self-care limitations 


timed-up-and-Go test; 


(UPDRS-II) 
17. Bielderman | Netherlands Older persons | 193 5 5 
[50] 
Outcome: Other model variables: Social functioning Mobility variable: Leg 
CASP-19 (partner, loneliness, social network, social strength, aerobic 


support), depressive symptoms, self-efficacy, 


socioeconomic status 


endurance, dynamic 
balance 
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Table 5.2. (continued) 


Study # / First | N Model Mobility 
Author Country | Population | Number | Variables Rank 
18. Mayo [51] | Canada | Older HIV 707 |18 7 

| | men | 
Outcome: | Other model variables: Latent variables for: Mobility variable: 
Single QOL Biological variables, symptoms, function, health | Physical function 
item perception, personal and environmental factors 

following Wilson-Cleary model | 


ADL Activities of Daily Living; MCS Mental Component Summary (SF-36 / RAND-36); OA 
Osteoarthritis; PCS Physical Component Summary (SF-36 / RAND-36); PDQ(L) Parkinson's 
Disease Quality of Life; Physical Function: Subscale of 10 self-report items from SF-36/RAND-36 
(limitation in vigorous and moderate activities, stairs, walking short and long distances, bending, 
lifting, and self-care); SRH Self-rated Health; UPDRS United Parkinson's Disability Rating Scale; 
WHO World Health Organization 


A feature of these papers is that the impact of mobility is evaluated in the context 
of other variables including personal factors, environmental factors, symptoms, and 
other activities. However, a limitation of these studies is there is no universally 
accepted measure of QoL and the outcomes modeled covered domains that are part 
of, but not, QoL such as physical health (often the PCS from the SF-36), general 
health, mental health, participation, illness intrusiveness, or generic or condition 
specific profile measures of HRQL. Two constructs closer to QoL were life- 
satisfaction and well being. Only two of the 18 studies measured of QoL, one with 
a QoL measure, CASP-19 (Study #16, and one with a single item (Study # 17 from 
Table 5.2). The studies are ordered according to the rank of mobility in explain QoL 
outcomes. 

In these studies, mobility was measured in two ways, through self-reported limi- 
tations or difficulties with mobility related activities and through tests of physical 
capacity. For the latter, these measures included impairments affecting mobility 
such as strength, balance, and aerobic capacity and performance tests of mobility 
such as gait speed, Timed-up-and-Go, and 6MWT. 

The first seven studies had at least one component of the mobility variable as the 
most important. Of these seven studies, four were from general population samples 
or older persons (Study # 1,3,4,5) and three were from clinical populations (Study 
# 2,6,7). Five studies had mobility as the second most important variable (Study # 
8-12) and these were all of clinical populations. 

The two studies (Study 17,18) where mobility variables were not related to 
QoL were the same two studies that used actual QoL measures rather than measures 
of related constructs. These measures relate to QoL of the person. Study #17 used 
the CASP-19 which measures the extent to which the older person feels control, 
autonomy, pleasure and self-realization. Study #18 used a single QoL item in men 
living with HIV, a condition that does not affect mobility primarily. The outcomes 
in many of the other studies included aspects of the body's QoL for which mobility 
would have a stronger influence. 
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This new distinction between QoL of the person and QoL of the body is impor- 
tant because health care targets the body. Monitoring the body’s QoL could be a 
valuable way for people to know about their body and to reduce the impact of illness 
and life’s stresses on the body. Monitoring the body’s QoL could also be an effective 
health care surveillance strategy particularly if it can be done unobtrusively and in 
real-time. 

QoL of the person is best reflected through global QoL measures including those 
of life satisfaction, whereas QoL of the body is reflected in outcomes related to in 
health, participation, and illness’ intrusiveness, by, for example impairing mobility. 
The importance of this distinction could be profound, as the person, with assistance 
for a growing portfolio of technologies, is better placed to monitor the QoL of their 
body in real time and react or adjust accordingly. Also, it is recognized that QoL of 
the person has a profound effect on the body through stress reaction or, in general, 
behaviors aimed at managing negative emotions [52], that the person could be made 
aware of through monitoring QoL of the body. 


Using Mobility to Quantify QoL 


If we were to use mobility to quantify QoL, how much will we under or overesti- 
mate QoL and in whom and under what circumstances? While the WHO provides a 
definition of QoL, in 1978, Flanagan [9] identified 15 components of QoL which are 
listed in Fig. 5.2. 

Based on the literature [36, 53, 54] and the clinical and research experience of 
the authors, an estimated 1/3 of QoL of the person would be explained by mobility, 
with helping others and active recreation most affected and learning, understanding 
self, and creativity the least affected (see Fig. 5.3). 


Health Material 


Personal safety Learning 
comforts 


Creative Relationship x Participation in 


public affairs 


expression 4 Helping others 


Understanding 


Socializing Leisure Soli 


Fig. 5.2 Flanagan’s (1978) Components of QOL 
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m Mobility mQOL 


Active recreation 
Helping others 

Health and safety 
Socializing 

Relationship with spouse 
Passive leisure activities 
Work 

Children / grand children 
Involved in public affairs 
Close friends 
Relationships 

Material comforts 
Expresing self creatively 
Understanding yourself 
Learning 


0 20 40 60 80 100 


Fig. 5.3 Amount of QOL explained by mobility 


What Mobility Parameters Can Be Monitored 
Via Technologies? 


Table 5.2 provides a list of mobility parameters that people have linked to QoL. Starting 
at the most global is life-space mobility [55], which relates to how far from home a 
person moves and has been shown to be associated with social support, capacity to 
drive, and gait speed [56]. One can tell a lot about a person through this one measure, 
which can be monitored through Global Positioning System (GPS) technology (or 
some combination of Cell-IDs [57]) that comes with most smart phones or smart wear- 
ables. Second on the list, and also very telling about the body’s QoL is the amount of 
vigorous activities carried out which can be monitored using heart rate and physical 
accelerations. Increased heart rate without movement would indicate stress (physical or 
emotional), fear, or cardiac pathology. Amount of time spent in activity would be 
another very important indicator of the body’s QoL and this is easily monitored using 
simple wearable devices that monitor accelerations. A common metric is sedentary time 
[58], which is known to be detrimental to health. Wearable accelerometers also provide 
information on speed of movement such as step cadence and duration of activity bouts. 
Canadian Physical Activity Guideline [7] for adults advocate a minimum of 150 min- 
utes of moderate activity accumulated over a week in bouts of 10 minutes; walking at a 
cadence of 100 steps a minute would meet this recommendation [59]. 

Higher levels of physical activity such as is achieved through climbing stairs can 
also be tracked on many smart phone/watch applications. For example a 4 hour 
game of golf on a hilly golf course that the senior author (NM) regularly walks 
accumulates 20,000 steps and the equivalent of 38 flights of stairs. 

Gait speed while considered the sixth vital sign [60] is actually less important 
than cadence and more difficult to track as it requires a measure of distance not just 
stepping frequency. Gait speed is easier to measure clinically where a fixed course 
can be walked and timed, raising its importance as a clinical indicator because of 
ease of measurement. The importance of gait speed for safety cannot be disputed as 
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people who cannot walk a speeds of greater than 0.5 m/sec (i.e., in terms of their 
capacity) are at risk of falls and benchmarks of 20.7 m/sec and > 1.0 m/sec indicate 
safety risk when crossing 2- and 4-lane streets [61]. However, the frequency at 
which people walk at different cadence bands a more relevant indicator of how the 
body is doing in everyday life. Tudor-Locke [59] showed that despite having capac- 
ity to walk at a health promoting pace when tested clinically, it is rare for the North 
American senior to do this in the real world for more than a few minutes a day. Mate 
et al. [62] showed the same was true for people with Multiple Sclerosis in that they 
do not reproduce what they are capable of doing on a clinical test in their real world 
environment, except of course if they lack capacity. 

Another clinical test that shows importance for QoL is the Timed-Up-and-Go 
(TUG) test [63]. This test is indicative of mobility as it requires standing up from a 
chair, walking 3 meters, turning around, and return to sit back down on the chair. 
Again, this test is easily done in the clinic, as it requires only a standard chair and 3 
meters of walking space. In the real world, the number of transitions from sitting to 
standing can be captured, as it is another metric available on standard accelerometers. 

These mobility measures relate to activity but it is also possible to track motor 
impairments that lead to mobility limitations such as slowness of movements that can 
result from stiff joints, resting and intention tremors, poor posture, balance, and poor 
gait quality. Monitoring these impairments would require different technologies dis- 
tributed to different parts of the body, but all are possible. There are apps for tremor and 
balance that require the person hold a smart phone. Posture, balance and gait quality 
can be measured using inertial devices attached to the back [64] or to the shoe [65—68]. 
A selection of wearable devices for mobility and health are shown below (Table 5.3). 


Table 5.3 Examples of wearable technologies that can monitor the QoL of the body 


Garmin activity tracker UPRIGHT Posture trainer 


q 


Wearable oximeter (AARC) Gait analyser 


E 
Ex 


Heel2Toe™ PhysioBiometrics Inc. Sweat monitor 
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How do these measurements relate to QoL of the body? Research show that 
slowness of movement, poor and irregular gait, poor posture, lack of activity can 
indicate states of pain, fatigue, low mood, apathy, or anxiety [69-73]. 


Mapping the Future 


We are at the cusp of changing the way we think about monitoring and remediating 
[74] and technology is poised to empower people to take charge of their own QoL, 
including, or starting from, their body’s QoL. Given the emergence of today’s and 
tomorrow’s personalized and miniaturized technologies, the future of monitoring 
the QoL of the body is inevitable [18]. We envision that the individuals will wear an 
accurate, well designed smartwatch that monitors activity, heart rate, oxygen satura- 
tion, and tremor, amongst other variables. For people with specific health chal- 
lenges, their posture, stability, and gait quality would be also monitored by small 
unobtrusive sticky devices placed on their spine, shoulder, ankle or shoe. They also 
would receive updates on how they are doing, as their devices are continually con- 
nected to their smartphone. They will be able to see how their body reacts to internal 
and external stimulae and learn how to respond to signs of threat [18]. They will 
also be able to program these devices to provide engaging and effective feedback for 
optimal performance, literally stamping in good mobility habits, and influencing 
positively their QoL in the long term. 
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Introduction 


In 1979, the International Association for the Study of Pain defined pain as “an 
unpleasant sensory and emotional experience associated with actual or potential 
tissue damage or described in terms of such damage” [1]. Pain is a subjective expe- 
rience corresponding to an unpleasant situation that may be both physical and psy- 
chological. Acute and chronic pain are considered two distinct medical conditions. 
Acute pain is provoked by a specific injury or disease, generally lasts no longer than 
6 months, and goes away when the underlying cause is gone [2]. Chronic pain, on 
the other hand, is a long-term condition. When associated with a specific injury or 
disease, it is considered a disease itself because it outlasts the normal healing time. 
It may even arise from a psychological state with no biological cause and with no 
recognizable endpoint [2]. In patients with chronic pain, overall quality of life is 
diminished [3], and for some patients, this pain can persist for one’s entire life. 
Aspects of quality of life that are usually influenced by chronic pain include, but are 
not limited to, sleep, cognitive and brain function, mood, mental health, and even 
sexual function [3-6]. Moreover, the amount one's quality of life decreases is 
strongly correlated with the severity of chronic pain experienced [3], so that higher 
levels of pain lead to more significant reductions in quality of life. 

The impact of chronic pain on quality of life calls for a reliable and continuous 
approach to pain monitoring and evaluation in order to provide the best possible sup- 
port to patients. The aim of such an approach should be to assess patients' pain 
throughout their lives in a timely and accurate manner without the use of self-reporting 
and to help patients cope with their situations so that they can avoid a considerable 
decrease in their quality of life. For this reason, this chapter presents iSenseYourPain, 
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a system design that aims to gather continuous data about patients’ behavior in order 
to understand how their pain experiences influence their lives. For example, the sys- 
tem seeks to determine if changes in sleep occur by measuring the number of hours 
and times of day when one sleeps, along with data such as the number of steps taken 
each day and the amount of time spent outside. The data is collected using ubiquitous 
devices and personal sensors integrated into patients’ everyday lives. This way, 
patients do not need to carry intrusive, special medical devices to collect the informa- 
tion. By facilitating real-time assessment, the iSenseYourPain system is primarily 
designed to determine when increases in a patient’s pain occur while measuring daily 
life activities and recognizing patterns of activity and pain experiences. In addition, 
the system aims to provide feedback to help patients manage pain exacerbations when 
needed with the help of their physician or relatives. 

This chapter is structured as follows. Section 2 discussed the pain and its assess- 
ment via self-report methods, while Sect. 3, its assessment via devices. Section 4 
presents the iSenseYourPain design n choices and Sect. 5 concludes the chapter. 


Self-Reported Pain and Chronic Pain Assessment 


Chronic pain requires frequent patient follow-up focusing on how pain is triggered 
[3], how it fluctuates with different behavioral patterns, patients’ medication regimens 
and daily life contexts, and the ways pain influences overall health and life quality. 
Currently, the evaluation of chronic pain is generally based on the use of paper ques- 
tionnaires and scales for which patients provide self-reports at predefined time inter- 
vals responding to a set of questions concerning their symptoms in the previous days 
or months. The most commonly used scales are described in what follows. 

Boonstra et al. [7] have developed a scale model to evaluate pain based on the 
Numeric Rating Scale. A common challenge with this type of scale, however, is 
defining the different cut-off points for the mild, moderate, and severe pain catego- 
ries into which the scale is divided. The Visual Analog Scale (VAS) is another 
common method for quantifying the severity of pain. It is a continuous outcome 
measure consisting of a scale 100 mm in length ranging from 0 to 100 with low- and 
high-end points corresponding to no pain and the worst pain. The VAS is easy to 
administer and has been validated for both adults and older children. It has also 
proven to be a reliable and valid technique to measure acute pain in emergency 
departments [8]. Meanwhile, the Brief Pain Inventory is commonly used to mea- 
sure a patient’s pain intensity and how much this pain influences their ability to live 
their everyday life. It consists of two different categories, namely pain intensity and 
pain interference [9]. The Medical Outcomes Study Pain Measures is another 
questionnaire that evaluates pain according to intensity, frequency, duration, and its 
impact on behavior and mood [10]. The Oswestry Low Back Pain Disability 
Questionnaire and the Back Pain Classification Scale are tools used by research- 
ers and disability evaluators to evaluate low back functional disability or psycho- 
logical disturbance [11, 12]. The Pain and Distress Scale, another frequently used 
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tool, is a measurement of mood and behavior that may be associated with acute 
pain. It does not describe the severity of patients’ pain itself but rather the physical 
and emotional reactions that can be attributed to limitations caused by pain in daily 
activities, including increased anxiety, depression, and decreased alertness [13]. In 
pediatric populations, facial expression drawings or “faces scales” are a popular 
method of assessing pain severity. A variety of faces scales exist, each of which uses 
a series of facial expressions to illustrate a spectrum of pain intensity. Faces scales 
are ordinal outcome measures consisting of a limited number of categorical 
responses ordered in a specific pattern. Although the optimum design of the facial 
expressions is frequently debated, the literature suggests that face-based rating 
scales are the preferred method of pain reporting among children. The Wong-Baker 
FACES Scale in particular has been implemented in multiple pediatric settings for 
pain assessment [14]. The McGill Pain Questionnaire (MPQ) is used to evaluate 
and monitor the pain over time, even to determine the effectiveness of any interven- 
tion [15]. The Pain Perception Profile (PPP) uses four different points of view to 
describe in the pain experience of each patient [16]. 

Finally, the Chronic Pain Grade Questionnaire is a multidimensional measure 
that assesses two dimensions of overall chronic pain severity: pain intensity and 
pain-related disability [17]. 

Table 6.1 provides an overview of the different pain scales, including the items 
to be filled out by the patient in each scale, the recall period, and the number of 


Table 6.1 Recap of different scales used to measure pain in patients 


Number of | Type of pain Number of 
Scale name items Recall period output levels 
The visual analog scale (VAS) [8] 1 Acute and chronic 10 
pain 
Now 
The Brief Pain Inventory [9] 11 Acute and chronic 11 
pain 


Now, recent days 
and past weeks 


The Medical Outcomes Study Pain 12 Chronic pain 12 

Measures [10] Past four weeks 

The Oswestry Low Back Pain Disability | 10 Chronic pain 6 

Questionnaire [11] 

The Back Pain Classification Scale [12] | 103 Acute and chronic Checklist — 1 
pain 

The Pain and Distress Scale [13] 20 Acute pain 4 
Recent days 

The McGill pain questionnaire (MPQ) 20 Chronic pain | 

[15] Now 

Pain perception profile (PPP) [16] 37 Acute pain 4 
Now 


Chronic Pain Grade Questionnaire [17] 7 Chronic pain 10 
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output levels provided by the scale based on the type of pain (acute or chronic) 
being assessed. 

One of the aims of the assessment system that is proposed in this chapter is to 
realize a shift from a paper-based self-reporting approach to an automatic and ubiq- 
uitous one. Self-reporting requires time and cognitive effort to carry out and cannot 
be performed frequently, especially to the degree required for constant monitoring. 
The latter approach is based on the use of various ubiquitous devices that can con- 
stantly assess pain-related behavior changes. The iSense YourPain system, which is 
presented in Sect. 4, is pervasive, ubiquitous, and non-invasive, requiring no input 
from the patient beyond an initial calibration phase and no equipment besides the 
technologies and devices already used in one’s everyday-life, thus increasing patient 
acceptance. Before the presentation of the system, however, Section 3 provides a 
brief overview of how smart devices have been used for pain detection until now in 
order to contextualize the development of iSense YourPain. 


Pain Detection Using Smart Devices 


In the last decade, there has been an exponential increase in the number of mobile 
and ubiquitous devices in use, while a considerable number of self-monitoring 
applications have been introduced that aim to assess and improve individuals’ over- 
all quality of life. Several studies have pointed out that, for these solutions to be 
effective, they should be easy to use, customizable, and adaptable to the routine and 
lifestyle of each person, including their location, social interactions, and healthcare 
needs, while providing timely and personalized suggestions [18]. 

Given that chronic pain can have a severely detrimental impact on quality of life, 
several systems and platforms have been developed to provide support to patients 
suffering from it. One such platform is that of online peer-support forums. Several 
studies have been conducted that analyze their interactive use among patients with 
similar symptoms and diseases to promote the exchange of information and per- 
sonal experience, provide distraction, and facilitate social or peer support. Meta- 
analysis of several trials in which online forums were leveraged in patients’ care 
showed that patients who used such forums experienced a significant reduction in 
pain and anxiety, loneliness, and withdrawn behavior, as well as a greater willing- 
ness to return for treatment [19-21]. 

Besides online forums, the use of personal device systems is becoming increas- 
ingly common in the context of chronic pain. Kristjánsdóttir et al. [22] have devel- 
oped a smartphone intervention system for women with widespread chronic pain. 
The intervention involves one face-to-face session between the patient and a nurse 
and four weeks of written communication between the patient and their therapist 
through the device. In Kristjánsdóttir et al.’s study of the system, participants filled 
daily smartphone diary entries to support their awareness and reflect on pain-related 
thoughts, feelings, and activities. The registered diaries were made available to a 
therapist who provided personalized written feedback to the patient based on 
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cognitive-behavioral therapy principles. The results suggest that a smartphone- 
delivered intervention with diaries and personalized feedback can reduce “‘catastro- 
phizing" and prevent increases in functional impairment and symptom levels in 
women with widespread chronic pain following inpatient rehabilitation. 

Meanwhile, multiple applications have been developed that focus on the man- 
agement and assessment of chronic pain. Painometer [23] is an app that helps users 
assess pain intensity, including four different pain scales (including a faces scale) 
that can be used by patients to report pain experiences to their physician. Its main 
purpose is to encourage patients to report their pain and make the act of reporting 
more acceptable by offering a simpler and more accurate means of communication 
between patient and physician (see Fig. 6.1). The iCanCope with Pain [24] program 
is an integrated web and smartphone application for children and adolescents suffer- 
ing from chronic pain. The goal of the application is to address the self-management 
needs of adolescents with chronic pain by improving access to disease information 
and symptom-management strategies while providing functionality for the self- 
monitoring of symptoms, the setting of personalized goals, pain coping skills train- 
ing, chronic pain education, and peer-based social support. 

Other studies in pain management have investigated the implementation of less 
ubiquitous or pervasive methods such as the use of external sensors such as electro- 
cardiogram (ECG) or electroencephalogram (EEG) (as shown in Fig. 6.2). These 
devices have been used to collect data for biological values such as levels of oxygen 
saturation in the blood (SPO;), body temperature, heart rate (HR), heart rate 
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Fig. 6.2 Example of external sensor-based systems for pain assessment [25, 26] 


variability (HRV), and galvanic skin response (GSR) [26, 27], as well as to perform 
simple electrocardiograms (ECGs) [25, 28] to detect and predict migraines in 
patients. Despite their effectiveness, it is clear that such approaches are far from 
easy to adopt for routine evaluation and monitoring of individuals’ behavior met- 
rics. Such external sensors are generally more invasive than personal wearable sen- 
sors, which in most cases are capable of performing the same measurements. For 
example, the Holter ECG monitor that is used to measure HR (shown in Fig. 6.2) is 
much more intrusive than a simple smartwatch, despite the fact that smartwatches 
are capable of monitoring HR continually with nearly the same accuracy. 

As this section has demonstrated, current research and medical practices involv- 
ing the use of smart devices for pain detection are either based on self-reporting and 
thus do not assess real-time changes in patients’ pain experience, or they rely on 
external devices that are likely to affect or interfere in the everyday life of the 
patient. For these reasons, this chapter proposes a system designed to unobtrusively 
collect accurate and timely information about patients’ behavior and identify cor- 
relations between changes in patients’ everyday life activities and their experiences 
of pain. In contrast to assessments that are based on self-reporting, the system also 
aims to provide immediate evaluations and, eventually, to support patients before 
chronic pain dramatically impacts the quality of their life and the lives of those 
around them. 


iSense YourPain: System Criteria and Design Choices 


In this section, we outline the proposed iSense YourPain system’s main requirements 
and components, beginning with a discussion of previous research concerning smart 
devices that informs the system’s overall approach. As indicated earlier in the 
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chapter, the system’s main purpose is to use everyday devices to collect different 
types of data, analyze the data, and determine if a patient is experiencing a higher 
level of pain than normal and in which circumstances. 

The design of the system and its components is based on the results of previous 
research that highlighted how behavior, and the interactions between a user and 
several daily-life smart objects, are influenced by emotional states and moods [29, 
30]. Pain is not an emotion or feeling but a physical and/or mental state that may 
alter an individual’s usual behavior. Such changes in behavior can be measured 
using several common devices, such as smartphones and other personal ubiquitous 
devices. The acquired data may have varying levels of granularity and encompass 
diverse modalities, ranging from the number of steps taken during the day to more 
complex measurements such as sleep quality or HR variability. Ultimately, the 
method of collecting data from everyday ubiquitous devices to assess the behaviors 
and behavior changes associated with pain meets the criteria that were initially set 
for the system: namely, the realization of automatic and minimally intrusive assess- 
ments that leverage patients’ daily life environments. 

The design of the iSenseYourPain system is presented in Fig. 6.3 below. The 
design includes four main components: 


e A sensing component, which consists of a set of sensors and ubiquitous devices 
capable of collecting data about patients’ lives that are embedded in the patient’s 
environment (Fig. 6.3, left side); 

* A self-report component occasionally used by the patient to self-report pain lev- 
els (Fig. 6.3, left side); 
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* The core analytics component, which is hosted at a dedicated secure server and 
which analyzes the collected data to model and evaluate individuals’ behaviors 
and correlate them with their pain levels (Fig. 6.3, middle); 

* A visualization component for the physician and/or the individual's relatives that 
summarizes the patient's pain experiences and aspects of their behavior and indi- 
cates potential areas of focus for additional interventions or usual care (Fig. 6.3, 


right side). 


Data Sources and Collection 


The system uses the data it collects to qualitatively and quantitatively describe the 
patient's behavior and define objective metrics in order to identify patterns linked to 
the patient's experienced pain levels. Table 6.2 below indicates how data corre- 
sponding to the different aspects of daily life potentially influenced by chronic pain 
can be sampled and collected within the system. In addition to data collected auto- 
matically through devices, a questionnaire based on the Chronic Pain Based 
Questionnaire (CPBQ) [17] is used occasionally to collect subjective pain experi- 
ence data that is later correlated with data collected by the devices. 


Table 6.2 Aspects of daily life influenced by pain and corresponding data types and rates 


Aspects of daily 
life 


General activity 


Type of data sources 


Smartphone, hybrid 
home sensors 


Sampling rate 


Hourly, daily 


Other factors 


Highly variable between days, 
seasons, weeks 


Mood Smartphone, hybrid | Weekly, monthly Highly subjective, additional 
home sensors data collected via self-reports 

Walking ability | Smartphone, Daily Strongly influenced by pain 
wearables level 

Work activity Smartphone Daily, weekly High variability according to 
interactions, the subject, additional data 
wearables/sensors collected via self-reports 


Relations with 


Smartphone, hybrid 


Daily, weekly, 


Number of interactions and 


other people home sensors monthly time spent vary highly 
according to subject 
Sleep Smartphone, Daily Influenced by various factors: 
wearables Food, alcohol, physical 
activity, etc. 
Enjoyment of Smartphone Daily Highly subjective, additional 
life interactions, data collected via self-reports 
wearables/sensors 
Perceived pain Structured Varies, frequency Subjective 
levels (CPBQ questionnaire progressively 
[17D decreases 
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There are two important aspects of the design of the data collection methods: 
modularity and ubiquity. The collection methods are ubiquitous because all data 
sources are embedded within everyday life devices such as smartphones, wearables, 
the home environment, TVs, and appliances. This design choice aims to reduce the 
system’s intrusiveness, leading to higher levels of acceptance from the patient while 
lowering the influence of the system itself on everyday behaviors. On the other 
hand, the system’s modularity is essential to making it as scalable as possible and 
adaptable to the specific set of sensors or devices available to and used by each 
patient. As is explained in Sect. 4.2 below, each patient is characterized by a unique 
model corresponding to their everyday behaviors and pain-related behavior changes. 
These models are continually developed and evaluated based on the available sen- 
sors and devices and the data they collect. 

As indicated above, patients are also occasionally asked to self-report pain levels 
by filling in a brief questionnaire on their smartphone in a way that leverages the 
Experience Sampling Method (ESM [31]). This self-reporting is carried out fre- 
quently at the beginning of the system use period, as it is important for creating and 
calibrating the patient’s personalized model. The estimated duration of this initial 
calibration phase is one to three weeks, depending on the regularity of the patients’ 
usual behaviors (such as sleep and activities) and the extent to which pain affects 
these activities. Once the model is pre-trained, the frequency with which patients 
submit self-reports via ESM will decrease, as the behavior—pain model will require 
only occasional smaller adjustments. In addition to the behaviors incorporated into 
the patient-specific model, there are some unusual behavioral indicators that may be 
strongly connected with pain experiences, such as the patient’s staying a full day 
indoors or spending unnaturally long periods in bed. These behaviors are treated as 
“red flags” indicating a pain experience regardless of the patient’s personalized 
model. When such unusual behaviors are detected by the system’s technologies, the 
patient will be prompted to self-report so that the system can develop its pain model 
more accurately. 


Data Analysis and Modeling 


The core analytics component of iSenseYourPain is hosted at a secure server and 
focuses on providing analytics of the features derived from the system’s various 
data sources. A model specifically derived for each patient is used to correlate 
behavior with the patient’s pain levels. Table 6.3 provides examples of the features 
that can be derived from different data sources. 

As every patient changes their behavior in different ways in response to pain 
experiences, the proposed system aims to identify the variations from normal behav- 
ior that are associated with pain for each particular patient. For example, sleep sen- 
sors can be used to model when a patients’ quality of sleep decreases based on the 
number of hours the patient is sleeping, the number of times they wake up during 
the night, and the duration of time they spend awake. Similarly, patients 
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Table 6.3 Examples of possible features derived from different data sources 


Type of data Source | Features 

Smartphone Smartphone Use time, screen interactions (touch, placement, 

interactions strength of touch), time spent with different 

applications 

Home life Hybrid home sensors | Time spent inside/outside home, time spent in 
(for activity, light, different rooms, types of activities performed at home 
noise, etc.) 

Sleep data Wearables Sleep duration, timing of sleep during 24 h period, 
Smartphone quality of sleep (especially sleep interruptions), time 


of different sleep phases (potentially inaccurate) 
Smartphone use around sleep and wake-up time 


Physical Wearables smartphone | Number of steps, time spent walking or inactive, 
activity timing of activities during 24 h period 
Heart data Wearables Average HR, resting HR, time spent in different HR 
zones, HRV 
Social Smartphone Number of calls and messages sent/received 
interactions (including social media usage), people met during the 
day (indoor/outdoor meetings, types of locations 
| visited) 


experiencing pain may take fewer steps outside their house than they normally do 
while constantly moving inside the house and changing their position (e.g., sitting 
or standing) or moving from room to room, all of which are behaviors that can be 
modeled by the system. Patients may also have different ranges of HR or HRV and 
smartphone habits. As each patient reacts to pain in different ways, the iSenseYour- 
Pain system uses the data collected during the initial training phase to understand 
each patient's behavior and develop a model that corresponds to their specific 
behavioral patterns. 


Conclusions 


Chronic pain is considered a permanent medical condition. It may be due to a spe- 
cific disease or injury, or it can arise from a psychological state, both of which may 
lead to chronic pain conditions with no predictable endpoint [2] and an overall 
decrease in quality of life. In patients with chronic pain, the behavioral changes 
associated with pain experiences tend to vary. Currently, these changes are gener- 
ally assessed via self-reported measures that are infrequent, subjective, and mem- 
ory based. 

In this chapter, we have proposed the iSenseYourPain system design as a means 
to constantly evaluate the pain experienced by patients through assessment of 
observed behavioral patterns. iSense YourPain collects data about patients’ everyday 
behaviors and models relevant aspects of their daily life. Based on a specific model 
developed for each patient, it then assesses and predicts patients’ pain levels based 
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on their behaviors. The system can inform a patient’s physician about increased pain 
levels or inform their relatives about their potential care needs. 

The iSenseYourPain system entails a significant contribution to the development 
of quality of life technologies [32] and the potential use of everyday technologies to 
quantify different aspects of individuals’ lives [33]. Overall, implementing systems 
such as iSenseYourPain may facilitate the achievement of better life quality for 
patients and for those around them. 

Future studies will be conducted that focus on implementing the system, both on 
the side of data collection and on that of the development of the patient model. 
Moreover, we plan to investigate which aspects of patients' daily lives that can be 
measured with personal and ubiquitous devices are most representative of behav- 
ioral change associated with pain experiences in specific types of patients. 
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Chapter 7 A 
Technologies for Quantifying Sleep: giente 
Improved Quality of Life or 

Overwhelming Gadgets? 


Sirinthip Roomkham, Bernd Ploderer, Simon Smith, and Dimitri Perrin 


The Importance of Sleep 


Sleep is, broadly, a recuperative and restorative process. As behavior, it is character- 
ized by reduced activity, withdrawal, quiescence, and reduced responsiveness. Put 
simply, humans typically tend to move to a designated sleep space (such as a bed), 
lie down, close their eyes and lie still. Sleep is also a very complex neurophysiologi- 
cal process, of which only some features can be readily observed or measured [1], 
with associated complex changes in physiology, including reduced respiration and 
heart rate, changes in heart-rate variability, cyclical changes in muscle tone, 
amongst others. 

Humans are regarded as diurnal (day active) animals, and as adults they tend to 
achieve their major sleep episode during the night-time hours. This timing of sleep 
is largely governed by circadian processes, and particularly by the evening expres- 
sion of the hormone melatonin, together with a ‘homeostatic’ increase in the likeli- 
hood of sleep, the longer a person is awake. These two processes ideally function to 
promote regular, consistent, and sufficient sleep. However, the achievement of good 
sleep also depends on opportunity and environment. In the contemporary context, 
sleep is often disrupted by choice or by externalities. These can include preferences 
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around work, study, or leisure hours (including engagement in social media, gaming 
and other device use), education demands, the presence of illness and pain, stress 
and worry, environmental factors including noise and extremes of temperature, nat- 
ural and other disasters, and the suppression of melatonin expression by artifi- 
cial light. 

While there is robust debate around the idea of an “epidemic of sleeplessness’ 
[2], many people are not getting sufficient sleep. Current evidence-based recom- 
mendations for sleep duration [3] suggest that 7 or more hours of sleep is required 
by most adults, and that more than 9 hours might be needed by young adults. In 
contrast, the US Centres for Disease Control report that 30-40% of people habitu- 
ally achieve fewer than 7 hours of sleep [4], defined as short sleep duration, with 
almost 70% of high school students sleeping fewer than 8 hours [5]. Although nor- 
mal sleep duration can vary between individuals, sleep restricted to less than 6 hours 
per night has been associated with significant impairments in cognitive perfor- 
mance, vigilance, and affect [6]. 

The effects of sleep loss can be seen very acutely. For example, sleep restricted 
by just a few hours can be observed in objective performance the next day [7, 8]. 
However, sleep loss can also accumulate over nights, so that an hour less sleep each 
night over a week may be equivalent in effect to a full night of sleep deprivation. 
The impact of chronic partial sleep restriction may not be observed immediately, but 
may be seen many years later through increased health and mental health problems 
[9]. This possibility has been raised by data showing strong links between involve- 
ment in shift work and later long-term health consequences. This means that sleep 
needs to be understood across timescales ranging from a single night to decades, 
and the nature, level, and intensity of measurement needs to reflect those timescales. 

The two most common sleep disorders are insomnia and Obstructive Sleep 
Apnoea (OSA). It is possible that as many as a billion adults globally have 
Obstructive Sleep Apnoea, with a prevalence approaching 50% in some countries 
[10, 11]. The prevalence of insomnia may range from 6—10% by strict clinical defi- 
nitions, to over 30% for poor or unsatisfactory sleep [12]. The direct (medical and 
industrial) and indirect (social) costs associated with these sleep disorders, and with 
other forms of poor or disrupted sleep, are very high, e.g. over $680 Billion across 
just 5 OECD nations [13]. The understanding, identification, and treatment of sleep 
disorders have seen very rapid growth over the past decade. While current treatment 
approaches to these disorders are regarded as both effective and cost effective [14, 
15], this growth still represents a very high additional cost. Due to the high global 
burden of OSA and insomnia, health-care systems will face major cost and logistic 
challenges, and must adopt more effective and efficient diagnostic and management 
strategies so that the negative health impacts can be minimized. There is a specific 
and growing need for broader public health, individual health, early intervention, 
and prevention approaches to sleep health. 

Sleep is clearly not a unitary state. Instead, it has distinct dimensions including 
duration, quality, timing and regularity, each of which can have corresponding 
objective and subjective meaning. Further, sleep matters for daytime function so 
that constructs such as daytime alertness, clarity, speed, energy and satisfaction are 
also very important. This multidimensionality means that approaches to the 
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measurement of sleep by devices must also be multidimensional, or be understood 
to be constrained to one or more dimensions, in order to understand sleep meaning- 
fully. Another implication is that a measure optimized for measurement along one 
dimension may not function well to measure another. Rather than asking ‘how well 
does this device measure sleep’, we might ask ‘what is it about sleep that this device 
measures’. Sleep science is a relatively new field, and the parallel rise in personal 
miniaturized sensing technology suggests a great opportunity for increased mea- 
surement, and increased range and mode of measurement, to inform new under- 
standing of this complex state. 

Individuals are increasingly interested in their own sleep, mirroring increasing 
public interest and understanding of the role of sleep in overall health and wellbeing 
alongside nutrition and exercise. Quality of Life Technologies (QoLT) provide 
immediate feedback through smart devices that people can use in their daily life to 
assess and enhance their health and well-being [16]. Most sleep tracking devices 
appear to have achieved the fundamental aims of QoLT, for instance, in that they 
claim to enable individuals to quantify their sleep through apps or wearable devices. 
However, important questions remain. Are these devices addressing a need that can- 
not be met by existing methods? Can they provide an objective and reliable assess- 
ment of sleep? Are there potential pitfalls in the use of such devices? Our Chapter 
aims to cover all these questions. 


Why Are Existing Methods Not Adequate? 


Sleep disorders are defined by recognized clinical criteria such as the International 
Classification of Sleep Disorders, ICSD-3 [17], the International Classification of 
Diseases, ICD-11 [18], and the Diagnostic and Statistical Manual of Mental 
Disorders, DSM-V [19]. Meeting these criteria generally requires an overnight 
sleep study using polysomnography (PSG) to confirm the clinical diagnosis of a 
specialist sleep physician (e.g. for suspected OSA), or may require careful clinical 
diagnosis by a psychologist without reference to PSG (in the case of insomnia). 

Polysomnography (PSG) is regarded as the gold standard for diagnosis of spe- 
cific sleep disorders (shown in Fig. 7.1). PSG objectively quantifies sleep-related 
indices from a combination of electroencephalogram (EEG, brain states), electro- 
cardiogram (ECG, cardiac states), electromyogram (EMG, muscle tension), electro- 
oculogram (EOG, eye movements in sleep) and indices related to sleep disorders 
such as nasal airway pressure and flow, respiratory effort, infra-red video, sound 
recording, and blood oxygen saturation. Individuals typically spend one or more 
nights in a hospital or clinic sleep laboratory, where they are monitored throughout 
the night, although there is increasing uptake of home-based and limited channel 
studies [20]. 

While PSG remains the gold standard for assessment of specific sleep disorders 
and is an evolved medical technology, it has certain features that limit its use. One 
or two nights of assessment does not allow for longer-term tracking of change, and 
does not assess the habitual sleep of individuals in their natural sleep environments 
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Fig. 7.1 A polysomnography 
test (image courtesy of Prof. 
David Lovell) 


and schedules. PSG requires extensive resources including equipment, staff, and 
software. In addition, it is not always easy for someone to undertake a sleep study 
when needed as waiting times for access can be considerable [21]. The interpreta- 
tion of sleep data is another challenge. Variability in the use of established rule sets 
for visual scoring of the PSG results in unreliability in key PSG outcome indices 
(such as the Apnea-Hypopnea Index, AHI) both within individual scorers and across 
laboratories [22], even when a single rule set is adopted [23]. For this reason, auto- 
mated analysis of studies using Machine Learning have been proposed as a way to 
reduce scoring variability and error. While these analytical models increase the con- 
sistency of scoring, and perform well against expert visual scoring, no single model 
has achieved widespread acceptance. The promise of neural network models and 
more recent Artificial Intelligence models has been recognized [24, 25] as a way to 
increase efficiency and accuracy in sleep medicine, but also to provide a deeper 
understanding of sleep and circadian biology. 

A second key method used to understand sleep is through elicitation of self- 
reported experience. This approach is required to capture the qualitative or subjec- 
tive dimensions of sleep [26]. In some cases, this experience cannot be measured 
objectively (e.g. satisfaction with sleep), or is necessary to understand predictors or 
consequences of poor sleep (such as increased alcohol use, or poor work perfor- 
mance). This can be done through structured ‘sleep diaries’ or time use surveys [27], 
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or through clinical and non-clinical rating scales and other measures. These mea- 
sures can be used to complement objective measurements from PSG, or may stand- 
alone for use in other settings such as epidemiological surveys. 

Subjective assessment using instruments such as the Pittsburgh Sleep Quality 
Index [28] is seen as relatively inexpensive and quick, and for those reasons is often 
used as an initial screen for sleep diagnosis [29]. However, the use of paper-and- 
pencil measures in particular can lead to high transcription and scoring costs, sig- 
nificant user effort, and simple measurement error. There is typically a requirement 
for careful development of subjective measures along sound psychometric princi- 
ples, and a need for high quality normative studies (e.g. stratified by age, gender, 
socioeconomics or other factors) to inform meaningful interpretation. This work has 
been done in a number of recent initiatives, e.g. Yu et al. [30], but these measures are 
not in widespread use in sleep medicine. 

A final common method for sleep and circadian rhythm measurement is actigra- 
phy. This is typically in the form of a wrist-worn ‘movement watch’ based on accel- 
erometry or other movement capture methods and which can be used to measure 
sleep over extended periods in a naturalistic environment including the person’s 
home, work, or other settings. Generation of these devices have been widely used in 
the past two decades for sleep assessment [31]. 

The American Academy of Sleep Medicine (AASM) provides a guideline to 
establish clinical practice recommendations for using actigraphy in sleep medicine. 
The purpose of using actigraphy is not to replace the gold standard of sleep measure- 
ment but rather to assist in deriving helpful metrics for sleep disorder assessment and 
treatment [32]. Actigraphy devices are typically continuously worn for 24-hours a 
day for several days to months. Sleep parameters are extracted from the movement 
data via specific sleep detection algorithms [33], and other derivatives such as circa- 
dian rhythm parameters and estimates of activity level can also be generated. By 
using actigraphy, clinicians are able to obtain unique information about sleep in a 
person’s natural or habitual sleep environment. This can be particularly important 
when sleep schedules are dictated by work demands (e.g. shift work), circadian 
phase shifts (e.g. international travel) or other circumstances where variation over 
time is predicted. Although actigraphy has been well studied and has been validated 
against PSG in specific populations, there are essential limitations to the concor- 
dance between the two approaches. The devices used in research and clinical practice 
tend to prioritize reliability, standardization, long battery life, and capacity for re-use 
over other considerations that might be more important for consumer-grade devices. 


Methods and Tools for Objective and Quantitative Assessment 
of Daily Sleep 


Consumers now have access to a number of technologies to objectively and quanti- 
tatively assess their daily sleep in their home environment. Below we review mobile 
applications, wearable devices (smartwatches, rings, headbands), as well as co- 
called ‘nearables’ that are near the individual, embedded in the sleep environment 
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(e.g. in mattresses). Broadly speaking, mobile applications are more widely avail- 
able to many consumers. Wearables and nearables, on the other hand, are more 
expensive; yet potentially provide a holistic view of our health and well-being along 
metrics such as daily activities, heart rate, and sleep. Wearables typically also con- 
tain a wider variety of sensor technology such as Electrocardiogram (ECG), 
Electroencephalogram (EEG) and Photoplethysmogram (PPG), which can lead to 
more accurate data. 

Mobile applications are readily available at the consumer’s fingertips. Consumers 
can manually record their sleep via sleep diaries or use mobile-embedded sensors to 
track their sleep automatically. Sleep logs and diaries allow consumers to record 
their bedtime, number of awakenings, activities before going to sleep and after 
awakening. There are two main approaches to keep track of sleep diaries: paper and 
electronic sleep diaries. Tonetti et al. [34] suggested that both methods achieved 
similar results, whereas electronic sleep diaries provide more benefits over paper 
diaries in terms of reducing time for data entry and automatically recording the time 
when the diary is logged. Choe et al. [35] emphasized that electronic sleep diaries 
are more beneficial to individuals when the diary application is quick to use, engag- 
ing, and encourages self-reflection. 

Automatic sleep tracking apps require minimal or no data entry. They are easy to 
use and inexpensive for self-tracking. However, their accuracy is often neglected, 
and their sleep results are often over claimed. Most sleep apps are not clinically vali- 
dated against laboratory PSG, the gold standard of sleep tracking, because PSG is 
labor intensive, expensive and difficult to access. Sleep Cycle, a popular sleep- 
tracking app, monitors a sleep-wake stage through motion or sound sensors, and 
provides a summary of sleep quality. Fino et al. [36] showed that the Sleep Cycle 
app failed to show adequate reliability when compared against PSG for sleep-wake 
detection. Despite the lack of reliability, Sleep Cycle app has continuously been 
used widely due to word of mouth (e.g. Editors’ Choice on the Apple App Store in 
October, 2020), and its potential benefits. Robbins et al. [37] report the use of Sleep 
Cycle to understand sleep duration and quality for four-year trends in general popu- 
lation. The results reveal helpful information. However, there is still some concern 
about its reliability and validity of sleep outcomes. Similar to Sleep Cycle, Sleep 
Time app shows poor correlation with PSG in terms of sleep parameters (e.g., sleep 
duration and number of awakenings) and sleep-wake stages [38]. Overall, further 
studies are needed to assess applications’ utility and examine how much trust that 
we could potentially improve so that we are more confident using it. 

Popular smartwatches like the Apple Watch, Samsung Gear and Fitbit become 
part of our daily living and are potentially helpful to remind us to do more exercise 
or to sleep more. Current studies [39, 40] investigate the accuracy of using the 
Apple Watch to determine sleep-wake stages and their results yielded reliable per- 
formance compared with a clinical-grade device and PSG. Roomkham’s study sug- 
gested that using an Apple Watch to monitor sleep-wake stages could be an add-on 
to traditional actigraphy as well as a way to study the broader population. Both 
studies have transparent methods to evaluate the reliability of smartwatches, which 
increases the confidence in using such devices or adopting their techniques for 
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tracking sleep in medicine, research, and self-tracking. There is clear potential for 
smartwatches to monitor not only sleep itself but also sleep quality [41]. 

Smart rings are becoming less intrusive and more attractive for consumers. Smart 
rings usually contain similar sensors to smartwatches, such as motion, heart rate, 
pulse rate, and body temperature. A recent investigation of healthy adolescents 
shows promising sleep outcomes beyond the sleep-wake stage into the whole sleep 
cycle [42]. However, the OURA ring used in this study employs a proprietary sleep 
algorithm to derive sleep stages, and most studies can only validate the overall sleep 
outcomes against PSG. 

A variety of headbands exist that track sleep through EEG sensors around the 
head. At the time of writing, the Dreem headband is a popular choice amongst con- 
sumers and researchers. The Dreem headband is able to measure brain activity 
(EEG), breathing, and movement. The results of heart rate, breathing frequency and 
respiratory rate variability are reliable which resulted in providing precision sleep 
stages and sleep parameters [43]. 

Nearables [44] are another type of consumer-grade device. They are placed near 
the sleep environment, e.g. under the mattress or on the bedside table. An example 
of nearables is the Beddit Sleep tracker, which measures heart rate through ballisto- 
cardiography (BCG) derived from pressure sensors under the mattress. Tuominen 
et al. [45] reported that the accuracy of the sleep tracking is low and that consumers 
should be careful interpreting these results. Similar with other types of consumer- 
grade devices, total sleep time was overestimated, and wake after sleep onset under- 
estimated. These results were also based on healthy participants only, and the 
reliability of such devices for users with obstructive sleep apnea, insomnia, and 
other types of sleep disorders is unclear. Another example is the DoppleSleep [46], 
a contactless sleep system that is placed next to the bed and uses radar signals to 
gather movement, heart rate, and breathing data. It performed well for sleep-wake 
classification (~90% recall), and to a lesser extent on sleep staging (~80% recall for 
REM vs. Non-REM). The S+ system by ResMed is another nearable, which relies 
on ultra-low power radiofrequency waves to monitor the user’s movements and 
breathing. Schade et al. [47] evaluated it against PSG and actigraphy and reported 
showed good sleep detection accuracy (~93%) and lower wake detection accuracy 
(69-73%), as for most devices. It is worth noting that wake detection was better 
than actigraphy (at 48%), thanks to a higher accuracy in detecting wake before 
sleep onset. 

Overall, it is important that convenience does not come at the expense of the 
accuracy and usefulness of the information that can be derived from consumer- 
grade devices [49]. Recent work from Depner et al. [48] identified important met- 
rics for sleep using wearable devices, and circadian metrics that may influence sleep 
(such as a level of exercise, which can be captured passively through the wearable 
devices). Tables 7.1 and 7.2 capture important metrics that are essential for valida- 
tion against the gold standard. 

Another crucial consideration is that when the sleep algorithm is provided by the 
device manufacturer, it is often a proprietary black box. One consequence is that 
data consistency is at the mercy of a software update over which the users often have 
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no control. This can be a problem, especially for long-term tracking. The device 
may also suddenly disappear from the market. Users, in particular in a research 
context, should favor devices that provide access to the raw data and allow the 
deployment of open algorithms, and tracking methods that have been validated for 
a broad range of users (age, healthy vs. sleep disorders, etc.). 


Benefits, Limitations, and Potential Pitfalls 


There are many benefits for consumers from having relatively easy access to sleep 
data from apps, wearables and nearables. A major benefit is a better awareness of 
sleep. People often purchase devices for other purposes, e.g. to track steps, and 
sleep data comes as a bonus [50, 51]. Once people have access to sleep data, they 
are curious about what the data may tell them about themselves, and whether they 
can see any trends. Based on such data, sleep may become more of a priority because 
they become more conscious of how much sleep they actually get, as well as how 
much sleep they ought to get for a healthy night of sleep. Having an awareness of 
sleep and tracking sleep often goes hand in hand with goal setting, e.g. aiming for 
at least 7 hours of sleep. Many apps provide virtual rewards for getting enough sleep 
that can increase motivation [52]. However, the main challenge here is that (unlike 
with the number of steps walked) people cannot voluntarily control how much time 
they spend in a particular sleep stage. Another benefit is the ability to explore links 
between sleep and other quality of life data, e.g. to identify if you sleep more on 
days when you exercise [50]. Exploring such links in the data is particularly useful 


Table 7.1 Sleep. What a Wearable Should/Can Measure (Adapted from [48]). Legend: a green 
tick (V^) represents metrics a wearable can measure; an orange tilde (^) represents metrics that only 
some devices can measure; a red exclamation mark (!) indicates proprietary methods 


DE Sleep Quality ! EEG Physiology Respiration 
Time in bed Vv Sleep Onset Sleep Staging v! Heart Rate Blood Oxygen 
Latency v (HR) - Levels (SPO2) 
Total Sleep Wake After Sleep Sleep Spindles Heart Rate Respiratory rate 
Time (TST) Onset (WASO) Y Variability (RR) ~ & effort 
(Bed time, (HRV) ~ 
wake time) ⁄ 
REM Latency Y Sleep Wave Blood Nasal Pressure 
Activity Pressure 
(BP) 
Sleep Efficiency v Slow Oscillations Body Air Flow 
Position 
Periodic Limb Skin Snoring 
Movement Conductivity 
(GSR) ~ 
Apnea 


Hypopnea Index 
(AHI) 
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Table 7.2 Circadian Physiology. What a Wearable Should/Can Measure (Adapted from [48]). 
Legend: a green tick (V^) represents metrics a wearable can measure; an orange tilde (~) represents 
metrics that only some devices can measure; ared exclamation mark (!) indicates proprietary methods 


Behavioural Physiological Environmental (nearables ) 

Sleep onset time v Heart Rate (HR) v 24h light exposure pattern (intensity 
and wavelength) 

Wake time v Blood Pressure (BP) v Ambient temperature (also: noise, 
humidity, air quality) ~ 

Timing of physical activity — Skin Temperature Geographical location ~ 

Y (Temp) ~ 

Timing of energy intake: Core body temperature EMA/self-assessments of 

food, caffeine, alcohol v (CBT) v environmental aspects (bed partner, 
kids, pets) 


Timing of supplement and Skin Conductivity 
mediation intake (GSR) ~ 
EMA/self-assessments of 

physical/mental state 

aspects (pain, mood, etc.) 


for people with sleep disorders and other chronic conditions, who seek to identify 
reasons for their sleep problems and other health conditions [53]. Going one step 
further, some people benefit from experimenting based on sleep and other quality of 
life data. For example, members of the Quantified Self community—a worldwide 
community of self-tracking enthusiasts—report self-experiments with sleep. These 
include experiments to improve sleep, e.g. by changing their sleep patterns, environ- 
mental aspects like reducing the noise and light in their bedroom, or sleep health 
strategies such as reducing their caffeine consumption. Conversely, sleep data also 
allows for experiments on the impact of sleep on other lifestyle factors, e.g. if they 
feel more alert at work and in everyday life [54]. 

Studies of consumers tracking their own sleep highlight also several challenges 
and pitfalls. In collecting and organizing data, consumers face the challenge of 
working with potentially inaccurate data, e.g. because accelerometer data alone 
cannot reliably determine insights into sleep stages (without EEG data), and accel- 
erometer may also inadvertently respond to the movements of partners in the same 
bed [55]. Consumer devices that contain EEG sensors potentially provide more 
accurate data, but the downside is that they are often uncomfortable to wear and 
impede sleep, and as a result are not worn for longer periods of time [56]. Consumers 
also report that they lack information on triggers (e.g. why they went to bed late) 
and contextual information (e.g. their stress and wellbeing) that may help to explain 
sleep data [54]. Even when consumers track such information through other means, 
they may not have the skills and tools to confidently relate data from different 
devices and sources to establish relationships, cause and effect [50]. 

The interpretation of sleep data also poses several challenges to consumers. 
People often report a lack of time to revisit and interpret data [57]. Sleep data can be 
overwhelming or even stressful when the desired outcomes do not materialize [58]. 
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Particular information about sleep stages can be difficult to interpret, because we do 
not consciously experience sleep stages, nor is the relationship between objective 
sleep stages and subjective sleep quality clear to consumers [53]. A related chal- 
lenge is that sleep data and what might objectively constitute quality sleep (e.g. 
more than 8 hours of sleep), may not relate to how consumers experience sleep or 
how refreshed they feel when they wake up [56]. There appears to be a disconnect 
between the various scientific metrics for quantifying sleep (e.g. sleep efficiency 
and sleep stages) and the way people experience and understand their sleep, which 
is more aligned with sleep duration and how satisfied or alert they feel as a result 
[59]. Unfortunately, this disconnect is not helped by the fact that most consumer 
devices are black boxes that conceal how data is collected and processed, which is 
partly a result of the complexity of sensor hardware and algorithms used [56], but 
partly also a deliberate choice of companies to protect their intellectual prop- 
erty [60]. 

A final challenge for sleep-trackers is that seeing a problem in the data is not the 
same as finding an opportunity to change it [61]. If our fitness-tracking device tells 
us that we have not achieved our 10,000 steps goal, then we can at least in principle 
remedy this by going for an extended walk. On the other hand, if the same device 
tells us that we only get 5 hours of sleep or not enough deep sleep, then the oppor- 
tunity to change this is less clear [50, 62]. We can change the sleep environment 
(e.g. block out light with curtains) and we can also improve our sleep hygiene (e.g. 
reduce the consumption of caffeinated products) to increase the chances of getting 
more sleep, but we cannot voluntarily control sleep itself [53]. Furthermore, in mak- 
ing adjustments to our environment and sleep hygiene we often face external con- 
straints, e.g. we may not be able to adjust the start time for school or work to better 
suit our own sleep needs, nor do we have control over autonomous conditions like 
our mood, stress, and hormone cycle [50]. 

To help mitigate these challenges, sleep-tracking at home is often conducted in 
collaboration with others. Family members, and especially bed partners, can play a 
vital role in supporting sleep tracking. Parents and older siblings often need to set 
up technology, interpret data, encourage healthy sleep hygiene, manage medical 
appointments, etc. [63]. Health professionals continue to play a crucial role to inter- 
pret the data, ask questions to reflect on sleep, and to provide advice on what actions 
to take [64]. 

Coming back to our original question, we have shown that personal sleep track- 
ing technologies can improve quality of life by increasing awareness of sleep, and 
by offering opportunities to explore connections with other quality of life data and 
potential sleep problems. However, we have also shown that collecting data, making 
sense of it to take action can be difficult and perhaps even overwhelming, because it 
requires expertise on sleep and tracking technology. We hope that this chapter will 
provide a useful introduction to these issues to get the most out of quantifiable sleep 
data. However, we also add a note of caution: quantification is important, but sleep 
cannot yet be reduced to a collection of measures. As we have discussed above, 
there is a mismatch between scientific sleep metrics and our subjective experience 
of sleep. Quantifying sleep also invites people to search for signs of sleep problems, 
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instead of promoting good sleep health [59]. No matter how elaborate, measures 
will always only provide a partial picture of sleep. A complete picture requires com- 
plementary information from subjective assessments, a broad perspective (overall 
health, but also inputs and interactions with other Quality of Life indicators), and an 
understanding of the varying needs and concerns of different individuals over a 
life course. 
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Chapter 8 (R) 
Improving Work Capacity and HRQoL: gest 
The Role of QoL Technologies 


Joan Julia Branin 


Introduction 


Physical activity is closely linked with health and well-being; however, many 
Americans do not engage in regular exercise. Only one in five adults in the US 
meets the CDC physical activity guidelines of 150 min of aerobic activity and 
2 days of muscle strengthening activity per week [1]. This trend of inactivity 
increases with age and can interfere with an individual’s capacity to work. The con- 
sequences of these trends are that the average worker can no longer deliver a full 
day’s effort in a physically, psychologically, and cognitively demanding job. 
Degenerative diseases associated with inactivity and obesity are epidemic. The ben- 
efits of physical activity and fitness extend beyond job performance and work 
capacity and include longer life and enhanced quality of life. Fit workers are more 
productive, are absent fewer days, and have a more positive attitude toward work 
and life in general [2]. 

Physical activity and fitness can be quantified by leveraging the available, afford- 
able fitness technology. Fitness technology, including trackers and smartphone 
applications (apps), have become increasingly popular for measuring and encourag- 
ing physical activity in recent years. Such technology encompasses individual fit- 
ness trackers that can stand alone, a fitness tracker paired with a companion app, or 
an app that can be downloaded onto a smartphone without the need for an extra 
device. The fitness tracker market is currently thriving, with estimates of almost 1.5 
billion dollars in revenue last year alone [3] and is expected to increase to a five- 
billion-dollar industry by 2019 [4]. A 2013 analysis revealed that there are over 
41,000 health and fitness apps currently available to the public via iTunes (e.g., Map 
My Walk, Runkeeper, My Fitness Pal) [5] and over half of smartphone users report 
having downloaded such an app [6]. Other personalized QoL technologies exist to 
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assess and improve physical activity and work capacity such as heart rate monitors, 
sleep trackers, and smart scales, to name a few. 

These physical activity and fitness trackers and apps are among the increasing 
number of Quality of Life Technologies (QoLT) for the assessment or improvement 
of an individual’s QoL. QoLT leverage the increasing availability of miniaturized 
computing, storage, and communication sensor- and actuator-based, context-rich 
technologies that can be embedded within various personal devices, for example, 
smartphones and wearables. They rely on hardware technologies (i.e., devices or 
physical interaction elements) or software technologies (e.g., apps or web-based 
interfaces) or, most likely, a combination of both. QoLT on an increasing scale can 
be personalized to satisfy the intended needs of the user anywhere and anytime and 
can be used in a continuous and longitudinal yet minimally intrusive way, in the 
individual’s daily life [7]. Overall personalized, and miniaturized computing QoL 
technologies have the potential to be change agents for an individual’s physical 
work capacity and health- related quality of life and significantly impact public 
health, research, and policy. 

This chapter addresses (1) the factors associated with variations in work capacity 
and quality of life; (2) the state-of-art of personalized, miniaturized computing QoL 
technologies for measuring and improving individual work capacity; (3) the use of 
activity trackers to quantify work capacity; and (4) strategies to enhance use of 
Web-based and non-Web-based tools and fitness technology for behavioral change, 
health management, and rehabilitation interventions for the self-management of 
work capacity and enhancement of health-related quality of life across the lifespan. 
This chapter concludes with guidelines for the monitoring and evaluating of digital 
health technologies and eHealth interventions and suggestions for future develop- 
ment of tools for the assessment and remediation of working capacity. The research 
question that guides this literature review is: How does the use of self-management 
QoL technologies affect work capacity and reported health-related quality of life? 


Definition and Measurement of Work Capacity 


According to the WHOQOL theoretical model, working capacity is the facet of 
physical health that examines a person's use of his or her energy for work. “Work” 
is defined as any major activity in which the person is engaged. Major activities 
might include paid work, unpaid work, voluntary community work, full-time study, 
care of children and household duties. This facet focuses on a person's ability to 
perform work, regardless of the type of work [8]. 

In the past, many research studies on physical activity and work capacity have 
relied on self-report instruments. However, such subjective measures of activity can 
be highly unreliable, stemming from memory, social acceptability and other biases, 
revealing both higher and lower estimates than an objective measurement [9, 10]. 
Since then, several standardized instruments have been developed to measure work 
capacity. 
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One of these standardized instruments, The Work Ability Index (WAL, is an 
instrument consisting of seven items, which takes into consideration the demands of 
work and the worker's health status and resources and is used to assess the work 
ability of workers during health examinations and workplace surveys. The purpose 
of the WAI is to help define necessary actions to maintain and promote work ability. 
The validity and reliability of the WAI has been assessed in correlation analyses. The 
WAI and all its items have been shown to reliably predict work disability, retirement, 
and mortality [11]. More recently, the validity of WAI has been studied by Radkiewich 
[12] and WAT's test-retest reliability by de Zwart [13]. The WAI has become the 
standard for assessing work capacity in occupational health research and the daily 
practice of occupational health care and has been translated into 24 languages. 

Another instrument, The Work-ability Support Scale (WSS), is a newer tool 
designed to assess vocational ability and support needs following onset of acquired 
disability to assist decision-making in vocational rehabilitation. It is designed to be 
used both for people who are working, or, as a planning tool for those considering 
returning to work. The tool has 16 items across three domains of work functioning: 
physical/environment, thinking and communicating, and social/behavioral. Scores 
ranges from 1 for constant support to 7 for independence. There are also an additional 
seven items related to contextual factors outside the workplace that could affect work 
functioning. Its scoring accuracy and rater reliability has been supported [14, 9]. 

Work capacity can also be assessed through a Functional Capacity Evaluation 
(FCE) which evaluates an individual's capacity to perform work activities related to 
his or her participation in employment. The FCE process is a set of tests, practices 
and observations highly specific for a type of job for which the individual is being 
assessed, as it compares the individual's health status, and body functions and struc- 
tures to the demands of this job and the work environment. It can provide an accu- 
rate measurement of an individual's ability to perform critical work tasks. This can 
help to determine an individual's capability/ability to return to work or their employ- 
ability. An FCE is performed on a one-on-one basis and can last up to 4 hours. A 
well-designed FCE should consist of a battery of standardized assessments that 
offer results in performance-based measures and demonstrates predictive value 
about the individual's return to work. The FCE report includes an overall physical 
demand level, a summary of job-specific physical abilities, a summary of perfor- 
mance consistency and overall voluntary effort, job match information, adaptations 
to enhance performance, and treatment recommendations, if requested [15]. 

Two self-reports have been utilized to assess the physical aspects of work capac- 
ity in cross-cultural settings. One instrument, The International Physical Activity 
Questionnaire- Long (IPAQ-L) is a 27-item questionnaire for use by either tele- 
phone or self-administered methods that can be used to obtain internationally com- 
parable data on health-related physical activity in theses domains (domestic 
physical activity (PA), occupational PA, leisure-time PA, active transportation and 
sitting time) and intensities of PA (vigorous, moderate, and walking). The instru- 
ment has undergone extensive reliability and validity testing across 12 countries (14 
sites). IPAQ has high reliability and moderate criteria validity in comparison with 
accelerometers. Good test-retest reliability for total PA, occupational PA, active 
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transportation, and vigorous intensity activities was shown. The results suggest that 
these measures have acceptable measurement properties for use in many settings 
and in different languages and are suitable for national population-based prevalence 
studies of participation in physical activity [16, 17]. 

Another instrument, The Global Physical Activity Questionnaire (GPAQ) is a 
16-item test developed by WHO as an improvement of the International Physical 
Activity Questionnaire (IPAQ) for use in cross-cultural settings. It collects informa- 
tion on physical activity participation in three settings or domains (activity at work, 
travel to and from places, and recreational activities) and sedentary behavior. It 
assesses work-related abilities such as able to “perform work involve vigorous- 
intensity activity that causes large increases in breathing or heart rate like [carrying 
or lifting heavy loads, digging or construction work] for at least 10 minutes continu- 
ously” and “time spent walking or bicycling for travel on a typical day.” Studies 
have shown fair-to-moderate validity of the GPAQ in a self-administered format in 
German, French, and Italian [18]. 

A few questionnaires have been designed for wide-scale, population-based sur- 
veillance of occupational physical activity (PA) behaviors. The Occupational 
Physical Activity Questionnaire (OPAQ) is a seven-item survey that identifies the 
average time per week spent in occupational tasks, e.g., sitting or standing, walking, 
and heavy labor activities. The modifications made when designing OPAQ improved 
its reliability for persons with stable work patterns, but at the expense of poorer reli- 
ability for persons with more variable PA. OPAQ did not have superior validity to 
IPAQ. OPAQ showed moderate to high 2-week test-retest reliability and moderate 
criterion validity when compared with detailed occupational PA records. The valid- 
ity of the OPAQ is similar to other established occupational PA questionnaires [19]. 

In addition, certain occupations have developed work capacity tests to assess the 
specific demands of their required work. An example, The Work Capacity Test 
(WCT), is a family of tests to determine firefighters’ physical capabilities to per- 
form the duties of wildland firefighting and to meet National Wildfire Coordinating 
Group (NWCG) standards for wildland firefighters. There are three levels of tests 
known as the “pack test" (arduous), “field test" (moderate), and “walk test" (light). 
The Arduous Pack Test is intentionally stressful as it tests the capacity of muscular 
strength and aerobic endurance of firefighters. Considerable effort has been spent on 
validating the test to the work demands of US wildland firefighters, for whom the 
test displays content validity; however, work is still needed to verify its reliability 
and criterion and construct validity [20]. 


Research Studies on Variations in Work Capacity and QoL 


Studies have documented greater exercise and physical work capacity among peo- 
ple who are active compared with sedentary individuals [21, 22]. Individuals with 
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self-rated good health are more likely to be employed [21]. Participants who had the 
capacity to work graded themselves as having both better health and HRQoL than 
those with a non-capacity to work [21]. 

Physical activity in older adults not only improves their physical work capacity 
but also reduces the risk of chronic diseases, such as cardiovascular disease, stroke, 
obesity, and hypertension; improves cognitive and mental health; lowers the chance 
of falls; and helps maintain a longer independent life. While a great deal of variation 
exists, significant declines in physical work capacity have been reported between 
the ages of 40 and 60 years due to decreases in aerobic and musculoskeletal capac- 
ity. These physical declines can lead to increases in work- related injuries and ill- 
ness. Differences in habitual physical activity, among other factors, greatly 
influences the variability seen in individual physical work capacity and its compo- 
nents [23]. 

Chronic diseases have an enormous impact on the ability to work. Being able to 
work is particularly important for the quality of life of people with chronic diseases 
[24]. Among individuals diagnosed with chronic conditions such as multiple sclero- 
sis (MS), a majority suffer with fatigue, which strongly influences their everyday 
life. Flensner [25] found that individuals with MS who had the capacity to work 
reported significantly less fatigue compared to those with no capacity to work. 
Additionally, the level of work capacity was significantly higher among those par- 
ticipants who were less sensitive to heat, while those who were sensitive to heat 
showed significantly more often a non-capacity to work. This study lends support to 
some existing evidence of the beneficial impact of good health on work ability in 
patients with MS. 

The relationship between physical and functional capacity and quality of life 
among elderly people who have a chronic disease was demonstrated in a study by 
Oztürk. Oztürk [26]. found that there are differences among elderly female and 
male individuals with a chronic disease in terms of the number of chronic diseases, 
types of chronic disease, mobility level, functional status, and QoL. One difference, 
mobility level, is related to functional capacity and QoL particularly in females; the 
higher the mobility level, the higher the QoL in females Rehabilitation programs to 
improve physical and functional capability and participation in daily activities may 
improve quality of life. 

Participation in regular physical activity has been associated with better cardio- 
metabolic indices [27], skeletal health [28], and cognitive and academic perfor- 
mance [29] in young people aged 7-18 years. Conversely, doing little or no physical 
activity in youth has been related to poor health outcomes and decreased quality of 
life in adulthood resulting in extended medical care and associated costs [30]. One 
study showed that the cost of MS in Sweden is about €600 million a year, one third 
of which are indirect costs associated with loss of production [31]. 

Reduced capacity to work is a major cause of high medical care costs and per- 
sonal care costs in the later stages of a disease [30]. 
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The State-of-Art of Personalized Computing QoL 
Technologies for Measuring Physical Activity 


Types of Wearable Activity Trackers and Other Devices 


In the past, many research studies on physical activity and work capacity have relied 
on self- report and self-report instruments to assess exercise behavior and capacity. 
However, such subjective measures of activity can be highly unreliable, revealing 
both higher and lower estimates than an objective measurement. People may over- 
estimate activity intensity or time spent to present a favorable impression. Self- 
reports often focus on discrete activities such as going for a run or working out at a 
gym. Objective measurement of physical activity can record activity that may be 
missed in a questionnaire or self-report. This may be especially relevant for older 
adults who are less likely to go to a gym on a regular basis [32]. Thus, there is an 
increased interest in using objective activity monitors and other available, afford- 
able QoL technologies to quantify physical activity and work capacity. 

Activity trackers refer to sensor-based personalized wearable devices that auto- 
matically track and monitor various indicators of physical activity, such as steps 
taken, stairs climbed, duration and quality of sleep, pulse or heart rate, and self- 
reported calories burned. Newer devices feature an electrodermal activity (EDA) 
sensor to help measure your stress level, an ECG sensor to assess your heart rhythm, 
an SpO2 sensor to measure the amount of oxygen in your blood, and a skin tempera- 
ture sensor. Some include built-in GPS. Activity trackers synchronize this data with 
users’ personal accounts, ensuring easy access from any device by the user. Activity 
trackers provide relatively unbiased data about basic physical activities and have the 
advantage of boosting physical activity through the integration of empirically tested 
behavioral change techniques such as goal setting, self-monitoring, social support, 
social comparison, feedback, and rewards [33], in contrast to antecedent technolo- 
gies, such as pedometers. Self-monitoring and goal setting using activity trackers 
have been especially effective in promoting self-efficacy and physical activity in 
interventions to improve capacity for work [28]. Studies have shown that physical 
activity and fitness is increased using wearable activity trackers [21, 23]. 

Good fitness level may increase the physical capacity for work as well as enable 
one to operationalize the factors important for self-reported work capacity. For 
example, wearable activity trackers data may enable one to quantify the answer for 
the WAI factor of “work ability in relation to the demands of the job” or “estimated 
work impairment due to diseases.” As for the WSS scale, fitness trackers may enable 
one to quantify the “physical and motor" or “mobility and access", “community 
mobility” and even “stamina and pacing” factors. Also, activity trackers may be 
useful in quantifying physical capabilities and capacity for certain work functions. 


Fitness Trackers Fitness trackers are at the heart of the fitness technology move- 
ment and have broad appeal. Their main appeal is that they create a snapshot of an 
individual’s physical fitness, which can empower individuals to make healthy 
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changes that impact their ability to engage in everyday work and nonwork activities. 
According to 2020 PCMag [33], a few examples of trackers that do a little of every- 
thing are the Fitbit Inspire HR and Fitbit Charge 3. Easy-to-use and easy-to-read 
collected data, they are both excellent options for first-time users. If an individual is 
more interested in having a full-fledged smartwatch that includes fitness tracking, 
then the Apple Watch Series 5 (which does a bit of everything except measure sleep) 
or Samsung Galaxy Fit are the best options. With a smartwatch, an individual gets 
apps and a lot more functionality, such as the ability to send text messages from 
one’s watch. One major disadvantage is battery life. The best fitness trackers can 
last a week or more, but smartwatches usually need to be charged once a day. For 
runners, cyclists, and anyone else who is already invested in fitness and monitoring 
peak work capacity, the best option is a runner’s watch that doubles as an all-day 
fitness tracker such as the Forerunner 45. 


Heart Rate Monitors (HRM) HRMs read an individual’s pulse while working out 
or active and are usually either chest straps or watch-style devices. They are not 
meant for monitoring heart rate 24/7 for medical purposes; for that, one needs an 
HRM that is FDA- approved or the equivalent in another country. Polar’s heart rate 
monitors, including the Polar OH1, have excellent tools for finding heart rate zones 
as well as explaining the activity’s benefits for the heart and body. Some fitness 
trackers or running watches have a heart rate sensor built in and allow one to see 
their heart rate in real time. Some such as JBL’s Reflect Fit are headphones, which 
take the pulse from the ear. HRMs may enable better quantification of stamina and 
pacing contributing to increased work capacity (WSS). 


Smartphone Fitness Apps Smartphone fitness and exercise apps provide users 
with data to quantify physical activities such as their “physical and motor” capacity 
and “stamina and pacing” with the additional advantage of immediate data aggrega- 
tion and analysis. Users can then modify their behaviors or work activities accord- 
ingly. Additionally, dedicated smartphone- based apps may enable one to quantify 
the “sensory and perceptual skills” (WSS) required by the job. 


For free workouts, The Johnson & Johnson Official 7-Minute Workout app has a 
variety of workouts for people of all fitness levels and of different lengths which are 
great for people who are just getting started with exercise and for frequent travelers 
to use in their hotel room. 

Run-tracking apps, such as Runkeeper and Strava, use the phone (or a compati- 
ble watch or fitness tracker) to record a runner’s pace, distance, mileage, and more. 
Apps such as MapMyFitness can track non-sport activities—for instance, shoveling 
snow, raking leaves, or walking briskly. MyFitnessPal app is one of the best apps for 
logging the foods one eats to count calories and get a nutritional breakdown. Weight 
Watchers and Noom include access to coaches through their apps as well as com- 
munity aspects so the user will not be alone on your fitness journey. If one wants 
hardcore training by an MMA champion, Touchfit: GSP gives an individual a series 
of progressively harder videos by fighter George St-Pierre. Jillian. 
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Michaels has a similar app with video-based workouts and a diet plan. Then 
there are apps and sites that specialize in one type of class, such as ballet, barre, and 
yoga or where you get a personal trainer to work with you on a personal fitness plan. 
Additionally, these dedicated smartphone-based apps resources may assist in quan- 
tifying “work ability” (WAI) and “sensory and perceptual skills” (WSS) required by 
the job. 


Sleep Trackers Sleep may influence one’s stamina, pacing and physical capacity 
to work and thus is an important variable to assess. Sleep trackers can track the 
user’s sleep so that an individual can learn more about his/her sleep patterns. Most 
fitness trackers now include sleep tracking. There are also smart mattresses with 
tracking technology built in. The most advanced, such as the Sleep Number 36 
makes adjustments during the night that are tailored to the user. Or one could con- 
sider a smart pillow that plays white noise and detects snoring, such as the REM- Fit 
Zeeq Smart Pillow. 


Other Devices There are other aspects of daily life and its relation to potential 
work capacity that are worth tracking and operationalizing. Smart water-bottles 
track your water consumption and remind an individual to drink to minimize dehy- 
dration. Smart clothing gives feedback to runners about their form in real time, 
commenting on heel strikes, cadence, etc. 


Some fitness enthusiasts believe that the secret to maximizing their fitness poten- 
tial is in their blood. A service called InsideTracker will send a phlebotomist to 
one's home or office to collect a blood sample and send it to a lab. There is even a 
wearable DNAband that helps an individual choose groceries based on their DNA, 
but it is now only available in the UK. 


Evidence for the Use and Effectiveness of Activity Trackers 


Academic and industry research has shown that the use of activity trackers can 
increase physical activity through continuous monitoring of activity progress, moti- 
vational messages, social support, and many other empirically tested behavioral 
change techniques [34-36]. That, in turn, may influence the work capacity [2, 26]. 
Wearable activity trackers have been shown to be effective in measuring and 
increasing the types of physical activity (e.g., steps, distance, calories expended, 
heart rate) that lead to increase work capacity. Adults who started using wearable 
activity trackers have been shown to increase daily activity levels [21]. A 7-month 
study of 18 participants (aged 36 to 73 years) who were given a wearable tracker 
found that 16 participants continued to use it after 7 months. The benefits of use 
included weight loss, social connection, and increased activity awareness [37]. 
Participants aged 60 years and older who were given a tracker reduced waist 
circumference and increased step count during another 12-week study [35]. African 
American and Hispanic older female participants, who tested a newly developed 
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tracking device in a 7-week study, increased their physical activity level, lost weight, 
and lowered blood pressure levels [21]. Activity trackers have been found to be 
more effective than their predecessors, where sedentary female older adults who 
used digital trackers significantly increased their physical activity compared with 
those who used pedometers [21]. A tracker that delivered prompts via a short mes- 
sage service has also been found effective in increasing moderate-to-vigorous phys- 
ical activity among overweight and obese adults [38]. 

Activity trackers facilitate physical activity in both young and older adults and 
are particularly beneficial for older adults because of the protective power of physi- 
cal activity against diseases associated with older ages [36]. Despite the evident 
benefits of activity trackers for older generations, digital care today is more avail- 
able to younger populations, leaving older adults on the periphery of the industry 
[39]. As little as 7% of older adults owned an activity tracker in 2014 [40]. Although 
many adults are now aware of this technology and its increased popularity, this 
population still shows slow rates of adoption that depend on many factors, including 
activity tracker trial and price [41]. Almost 84% of older adults aged 65 years and 
older do not meet the aerobic and muscle-strengthening physical activity require- 
ments [42], which makes activity trackers a particularly relevant technology for this 
age group. Physical activity recommendations for older adults tend to focus on 
moderate-intensity aerobic and muscle-strengthening activities such as walking, 
jogging, bicycle riding, yard work, and gardening [43]. Some of these activities are 
tracked by wearable technology. 

Activity trackers have the advantage of boosting physical activity through the 
integration of empirically tested behavioral change techniques such as goal setting, 
self-monitoring, social support, social comparison, feedback, and rewards [36] in 
contrast to antecedent technologies, such as pedometers. Self-monitoring and goal 
setting have been especially effective in promoting self-efficacy and physical activ- 
ity in functional capacity interventions. Although wearing a new piece of health 
technology is a novel activity for older adults, they appreciate the activity tracker’s 
contribution to self-awareness and goal setting. Activity trackers provide older 
adults with relatively unbiased data about basic activities. In addition, older adults 
view activity trackers as helpful motivators in achieving walking goals and compet- 
ing with themselves [44]. 

Another important advantage of activity tracker usage in the 65+ population is 
social connection. For a population that is characterized by social isolation and 
loneliness, technology such as activity trackers that addresses social connectivity 
needs is perceived as helpful in overcoming barriers to increase physical activ- 
ity [45]. 

Although activity trackers can be helpful in increasing physical activity, this 
technology is not ideal. For example, in a study with 8 adults who were aged 
75 years or older, 3 participants experienced technical problems with the activity 
trackers, preventing them from gathering any activity feedback. Participants 
reported that they could only get the activity tracker to work 78% of the time [46]. 

Thus, the full potential of quantifying and leveraging the available, affordable 
fitness technology in enhancing the capacity to work has only begun to be realized. 
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Each year additional new and upgraded wearable devices and new apps with 
expanded tracking features such as monitoring blood oxygen saturation levels, feed- 
back on sleep quality and stamina levels, and anxiety and panic attack tracking are 
being introduced to enhance the ability of these devices to monitor and facilitate 
major improvements in physical work capacity. 


Validity and Reliability of Fitness Trackers for Measuring 
Physical Activity 


Many studies have tested the utility of fitness trackers for measuring physical activ- 
ity, with varied results [46, 47]. For each fitness tracker, there are many models, with 
a variety of algorithms that provide activity estimates. It is important to note the 
brand, model, and body placement used to facilitate comparisons across studies. 
Early studies found that fitness trackers including the wrist-worn Fitbit and Fitbit 
Ultra and the waist-worn Fitbit One have acceptable reliability and validity compa- 
rable to research-standard devices in the lab [47]. Van Remoortel. 

[47] conducted a systematic review to identify whether available activity moni- 
tors (AM) have been appropriately validated for use in assessing physical activity in 
healthy populations and in those with chronic diseases. The latter population walk 
more slowly than healthy individuals which is reflected in their six-minute walking 
distance. This review suggests that most monitors (and pedometers) are less accu- 
rate during slower walking speeds. Validation studies of activity monitors are highly 
heterogeneous which is partly explained by the type of activity monitor and the 
activity monitor outcome. Since activity monitors are less accurate at slow walking 
speeds and information about validated activity monitors in chronic disease popula- 
tions is lacking, proper validation studies in these populations are needed prior to 
their inclusion in clinical trials. 


From Activity Trackers to Quantified Work Capacity 


With respect to use of the evidence of fitness data and work capacity, wearable 
activity trackers data may enable one to quantify the answers for self-reports and 
self-report instruments. 

Good fitness level may increase the physical capacity for work as well as enable 
one to operationalize the factors important for self-reported work capacity. For 
example, the wearable activity trackers data may enable one to quantify the answer 
for the WAI factor of “work ability in relation to the demands of the job” or “esti- 
mated work impairment due to diseases”. As for the WSS scale, the fitness trackers 
may enable to quantify the “physical and motor” or “mobility and access”, “com- 
munity mobility” and even “stamina and pacing” factors. In addition, they may be 
useful in quantifying physical capabilities and capacity for certain work functions. 
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Evaluating the match between the physical, mental, social, environmental, and 
the organizational demands of a person’s work and his or her capacity to meet these 
demands are important in assessing work capacity. Measurement of workability 
requires consideration of a range of factors, including physical ability to perform 
tasks, ability to cope with the cognitive/communication demands of the job, and to 
function appropriately in the social and environmental context of the work. Wearable 
activity trackers can quantify many of these factors and are especially useful for 
individuals with physically demanding work such as a mail carrier/postman and a 
wildlands firefighter (see Table 8.1). 


Strategies to Improve Work Capacity Using Behavioral 
Change Techniques and QoLT Interventions 


Work capacity is the employee's ability to accomplish production goals without 
undue fatigue, and without becoming a hazard to oneself or coworkers. It is a com- 
plex composite of aerobic and muscular fitness, natural abilities, intelligence, skill, 
experience, acclimatization, nutrition, and, of course, motivation. Even the most 
highly motivated workers may fail if they lack the strength or endurance required by 
the job. For prolonged arduous work, fitness is the most important determinant of 
work capacity [2]. While a great deal of variation exists, an average decline of 20% 
in physical work capacity has been reported between the ages of 40 and 60 years, 
due to decreases in aerobic and musculoskeletal capacity. These declines can con- 
tribute to decreased work capacity, and consequential increases in work-related 
injuries and illness [23]. 

Variations in work capacity over time can be quantified and self-monitored using 
quality of life technologies. The increasing availability of miniaturized computing, 
storage, and communication sensor-and actuator-based, context-rich technologies 
that can be embedded within various personal devices, such as smartphones and 
wearables, offer the opportunity to increase physical activity and lessen some work 
capacity declines through the continuous measuring, monitoring, and modifying of 
lifestyle behaviors such as the type, duration, and intensity of exercise, nutrition, 
sleep quality, and water hydration [48— 50]. 


Use of Digital Fitness Devices and Behavioral Change 
Management Techniques 


One strategy to improve work capacity is by using personalized, wearable activity 
tracking and physical monitoring devices with behavioral change techniques (BCTs) 
and evidence-based behavioral interventions [51]. Some of the more common 
behavioral change techniques are goal setting, feedback, rewards, social support, 
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Table 8.1 Quantification of Work Capacity Using Wearable QoL Technologies 


Scale and Question/ 
Factor on 


Data Source to 


Examples: Mail 
Carrier/ Postman 


Standardized Quantify the Sampling | Notes, Other and Wildlands 
Instrument Factor Rate Factors Firefighter 
General use 
WAT: “Work ability in | Fitness trackers: | Daily Need benchmark | Delivery of the 
relation to the Steps, distance, to quantify mail to door either 
demands of the job” | ECG “demands ofthe | on foot with or 
job" without a mail 
buggy or 
motorized vehicle/ 
electric bicycle. 
Ability to lift 
heavy mail sacks. 
WAI: “Estimated Fitness trackers: | Daily/ Need benchmark | Having a limit of 
work impairment due | Steps, distance, weekly capacity to max number of 
to diseases” ECG, SpO2 depending | quantify km/ mi a day due 
on nature | “impairment” to illness. 
of illness 
WSS: “Physical and | Fitness trackers: | Weekly Need benchmark | Ability to bike 
motor” Steps, distance, report of motor skills and/or drive long 
ECG detailed required (e.g., distances and 
motor skills via lifting, dexterity, | stretches of time. 
accelerometer coordination, Dexterity and 
balance). ability to open 
different types of 
doors and 
manipulate a huge 
number of letters/ 
packages with 
high accuracy. 
WSS: “Sensory and Smartphone: Use, | Daily/ Need benchmark | Recognition of 
perceptual” tests, GPS, water | weekly of visual, where the post 
hydration levels auditory, and man is and 
perceptual skills | delivery location. 
required. Mapping mail 
route using GPS 
Navigational 
system degree of 
water hydration. 
WSS: “Mobility and | Fitness trackers: | Daily Need benchmark | Covering 
access" Steps, distance, of mobility skills | specified number 
smartphone: required. of km/mi a day 
Indoor/outdoor, (biking 
mobility type, walking). 


duration, exercise 
modality 
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Scale and Question/ 
Factor on 


Data Source to 


Examples: Mail 
Carrier/ Postman 


Standardized Quantify the Sampling | Notes, Other and Wildlands 
Instrument Factor Rate Factors Firefighter 
WSS: “Community Smartphone: Weekly, as | Need definition of | Covering 
mobility” Indoor/outdoor, needed community and specified number 
mobility type, mobility job of km/mi a day 
duration, exercise requirements. (biking+ 
modality walking), number 
of residences or 
businesses or 
geographic area a 
day 
WSS: “Stamina and | Sleep HRM, HRV | Weekly Self-reported Speed of biking or 
pacing” Distance per time energy and walking (app, 
(km/h or mph), fatigue levels. tracker) may also 
use speech 
recognition for 
energy 
assessment. 
FCE form “climb Fitness trackers: | As needed | Observation/ Climb stairs and 
stairs, ladders” “walk | Steps, distance, evaluation by steep driveways to 
on uneven ground” ECG, SpO2 independent third | deliver mail walk 
party. over uneven 
Standards vary terrain or over 
for return-to- objects to deliver 
work by job mail. 
category. 
Employment or 
union standards. 
IPAQ-L; Fitness trackers Daily/ Self-report Delivery of the 
“Vigorous activities” | steps, distance, typical job- related mail to door either 
“moderate activities” | ECG, minutes week physical activity | on foot with or 


and “walking”. “As 
part of your work”: 
Duration and number 
of times 


cardio levels 


levels in past 

7 days. 

Need benchmark 
of stamina and 
motor skills 
requirements. 
Established by 
employer or 
union 


without a mail 
buggy or 
motorized vehicle. 
Ability to lift 
heavy mail sacks 
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Scale and Question/ 
Factor on 


Data Source to 


Examples: Mail 
Carrier/ Postman 


Standardized Quantify the Sampling | Notes, Other and Wildlands 

Instrument Factor Rate Factors Firefighter 

Global physical Fitness trackers, | Daily/ Self-report Delivery of the 

activity steps, distance, typical job- related mail to door either 

questionnaire time, HRM, week physical activity | on foot with or 

(GPAQ). cardio levels levels in past without a mail 

"Perform work that oxygen saturation 7 days. Need buggy or 

involves vigorous- levels, SpO2, benchmark of motorized vehicle. 

intensity activity" ECG stamina and Ability to carry 

“large increases in motor skills heavy mail sacks. 

breathing or heart requirements. 

rate"" carrying or 

lifting heavy loads... 

for at least 10 minutes 

continuously” “time 

walking or bicycling 

for work 

Travel" 

Physical activity Fitness trackers, | Monthly Need benchmark | Ability to deliver 

assessment tool steps, distance, to quantify mail, walk 

(PAAT) time, HRM, ECG demands of the distances. 

«been regularly work vs non work 

physically active for activities 

the last 1-5 months" 

OPAQ: “Hours walk | Fitness trackers, | Weekly Need benchmark | Ability to stand to 

at work" "hours sitting | steps, distance, to quantify deliver mail and 

or standing during HRM, ECG, specific demands | walk distances; 

work" “hours perform | accelerometer, of the work carry heavy mail 

heavy labor activities sacks. 

at work" 

Specific use 

WCT: (job Fitness trackers: | Daily, as Pack test Example: 

Specific test used by | Steps, distance, required benchmark of Wildlands 

wildlands firefighters) | water hydration, 3-mile hike with a | firefighter 

WCT: “Pack test" HRM, HRV, 45-pound pack Ability to perform 
ECG, SpO2 over level terrain | arduous work 


in 45 minutes, 
approximates an 
aerobic fitness 
score of 45--the 
established 
standard for 
wildland 
Firefighters. 


over prolong 
periods of time. 
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Scale and Question/ Examples: Mail 
Factor on Data Source to Carrier/ Postman 
Standardized Quantify the Sampling | Notes, Other and Wildlands 
Instrument | Factor Rate Factors | Firefighter 
WCT” field test” Fitness trackers: | Daily, as Field test Ability to perform 
Steps, distance, required benchmark of moderately 
HRM, ECG, 2- mile hike with | strenuous duties 
SpO2 a 25-pound pack |e.g., considerable 
in 30 minutes. walking over 
The passing irregular ground, 
score, standing for long 
approximates an | periods, lifting 25 
aerobic fitness to 50 pounds. 
| | | score of 40. | 
WCT “walk test” Fitness trackers: | Daily, as Walk test | Ability to meet 
Steps, distance, required benchmark of 1 emergencies and 
stress mile in evacuate to a 
16 minutes with a | safety zone. 
load, Approx. An 
aerobic fitness 
score of 35. 


Note: WAI The Work Ability Index; WSS The Work-ability Support Scale; FCE Functional 
Capacity Evaluation; [PAQ-L International Physical Activity Questionnaire-Long Version; GPAQ 
Global Physical Activity Questionnaire; OPAQ Occupational Physical Activity Questionnaire; 
WCT Work Capacity Test 


coaching, identifying barriers/problem solving, and action planning. Self-regulatory 
behavior change techniques such as goal setting, self- monitoring, and social sup- 
port have been associated with greater increases in physical activity than those that 
did not [32]. The extent that fitness trackers can enhance physical capacity to work 
may be dependent on goal setting such as the daily step goal of 10,000 steps or 250 
steps within an hour. Feedback and rewards, another set of behavior change tech- 
niques closely tied to goal setting, include the activity tracker reminders, text mes- 
sages, and real-time alerts when the user has met a goal or has been sedentary for 
too long which have been reported to be effective in motivating and increasing 
physical activity and fitness [52, 53]. Social factors such as social support or com- 
petition have been shown to increase engagement, adherence, and completion in 
physical activity interventions beneficial to modifying work capacity [52, 54]. 
Coaching is often used in fitness technologies and interventions to motivate 
increases in physical activity through weekly information sessions that encouraged 
healthy behaviors and a weekly 30-min group walking session [52]. BCT tech- 
niques used with fitness technology can lead to increases in physical activity, endur- 
ance, muscle strength, and decreases in physical work fatigue [32]. 

However, certain BCT strategies may be more effective for different age groups 
in enhancing the physical aspects of work capacity using wearable personal activity 
devices. A critical analysis by Mercer [54] noted that self-regulation techniques 
present in wearable activity trackers such as goal setting, feedback, and social 
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support are effective for younger adults, whereas older adults may benefit more 
from problem-solving, rewards for successful behavior, and modeling or demon- 
strating behavior. 

Some of these behavioral change techniques are included in fitness technology. 
An analysis of seven commercially available fitness trackers (Jawbone UP24, Nike 
Fuelband, Polar Loop, Misfit Shine, Withings Pulse, Fitbit Zip, and Spark) revealed 
that most or all of these trackers included goal setting, feedback, rewards, self- 
monitoring, and social support [54]. Many fitness trackers include a “mobile coach” 
within their connected application or website for encouraging increases in physical 
activity, stamina, and endurance. More recently, Lyons [36] used BCTs to system- 
atically analyze 13 wearable activity trackers and found that all of these trackers 
helped users to self-monitor behavior, obtain feedback on behavior, and add objects 
to the environment while also generally support users in goal setting and comparing 
their behavior with their goal. However, many health and fitness apps do not include 
BCT strategies thus their impact on the physical activity and the strength, and 
endurance aspects of work capacity may be limited [32]. 


Challenges Using Quality of Life Technologies 
in the Improvement of Work Capacity 


Several challenges exist in using quality of life technologies in the improvement of 
work capacity. One challenge for the use of wearable activity trackers for BCT for 
physical activity and work capacity interventions is that there is no guarantee that 
the user will encounter a technique even if it is present on the device. Another chal- 
lenge is that trackers are complex tools with multiple features and different users are 
likely to have different experiences. Similarly, a user who has a lower health or 
technology literacy may not explore the features as deeply as a health professional 
or expert technology user, regardless of age. Thus, the potential of wearable activity 
trackers to increase fitness behaviors through BCTs leading to improved capacity 
for physical activity and work can vary greatly. 


Discussion and Recommendations 


The Quantified Self Movement (QS) is a relatively recent trend wherein QS prac- 
titioners rely on the wealth of digital data originating from wearables, applica- 
tions, and self-reports to enable them to assess diverse domains of daily 
life---physical state, (e.g., mobility, steps), psychological state (e.g., mood), social 
interactions (e.g., number of Facebook “likes”) and environmental context they 
are in (e.g., pollution) which contribute to an individual’s Quality of Life (QoL). 
The collected QS data enables an individual’s state and behavioral patterns to be 
assessed through these different QoL domains, based on which individualized 
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feedback can be provided, in turn enabling the improvement in the individual’s 
state and QoL [55, 56]. 

Continuous monitoring is at the heart of the Quantified Self movement since it 
often gives a more accurate picture of human motion realities than short periods of 
laboratory testing and may provide a more accurate assessment of individual work 
capacity across different time periods and work-related activities. The long-term 
vision of QS activity is that of a systemic monitoring approach where an individu- 
al’s continuous personal information climate provides real-time performance opti- 
mization suggestions. The availability of powerful, personalized, and wearable 
mobile devices facilitates the provision of ubiquitous computing applications that 
enable clinicians and employers to better assess and monitor work capacity in the 
workplace and everyday life and to develop more effective interventions to enhance 
the physical and emotional aspects of capacity to work than previously possible. 

While the concept of the quantified self may have begun with self-tracking at the 
individual level, the term is quickly being extended to include “group data” and the 
idea of aggregated data from multiple quantified selves as self-trackers share and 
work collaboratively with their data. Using QS group device data could be helpful 
in quantifying potential healthcare cost reductions and savings and for verifying 
user behaviors in behavioral change management programs in the public and private 
sectors. QS group data has the potential to benefit society by going beyond data 
creation and information generation to meaning-making and action-taking strate- 
gies with applications for impacting work capacity on the population level. 

However, more research on the effectiveness of QS monitoring with QoL tech- 
nologies is needed. To date, relatively few high-quality studies have been conducted 
examining the correlation between physical activity and work capacity and the over- 
all effectiveness of physical activity trackers and smartphone interventions. Future 
studies should describe these interventions, the device and app features, and the AI 
and algorithms used with adequate detail so results can be reproduced, and lessons 
learned to advance work capacity and QoLT research. 

Future research should also be directed toward enhancing an understanding of 
the time course of intervention effects in enhancing one’s capacity for work. Of 
particular interest is a better understanding of the timepoint at which peak effect 
size is reached, the timepoint at which user engagement decreases, and the factors 
that underpin these phenomena. The relatively short- term nature of positive effects 
suggest that additional efforts are required to design app features which help sustain 
user engagement with the app over time, perhaps through modules, unlockable con- 
tent, gaming, and rewards. Sustaining user engagement is particularly important for 
smartphone-based interventions quantifying the physical aspects of work capacity 
due to the absence of human support and minimal supportive accountability. 

Mobile technology used for assessing and monitoring work capacity should con- 
tinue to emphasize ease of use, function, feedback, tailored information, ability to 
personalize design, and design-aesthetic as highly ranked engagement strategies. It 
will be useful for future app designs to incorporate long-term engagement strategies 
as increased exposure to the intervention can lead to larger, longer lasting effects in 
improving physical fitness and in turn, influencing work capacity. 
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Fitness technology should continue to include theoretically derived behavior 
change techniques that are useful for their intended population. Strategies such as 
goal setting, self- monitoring, feedback, rewards, social support, and coaching seem 
to be especially helpful in increasing activity and healthy behaviors that impact the 
work capacity of younger adults; while older adults may benefit more from problem- 
solving, rewards for successful behavior, and modeling or demonstrating behavior. 
The use of QS technologies as different types of affordances supporting the goal- 
oriented actions by individuals can, in turn, improve capacity to work and their QoL. 

Regardless of the type of intervention, efforts should be made to address the bar- 
riers that keep inactive people, especially older adults and low-SES populations, 
from using wearable activity devices to better help them engage in regular physical 
activity resulting in decreased sedentary behaviors and increased work capacity 
especially work requiring physical fitness and endurance. 

It is recommended that app and mobile device developers and behavior change 
experts collaborate when developing apps used to monitor physical activity, physi- 
cal fitness, or the physical aspects of work capacity. By understanding app usage, 
guidelines can be developed to create apps based on health behavior research to 
better quantify and promote long-term physical activity and sustainable work 
capacity. 

Lastly, the use of digital, mobile, and wireless QoL technologies and tools in an 
evidence-based, structured environment can lead to a fundamental transformation 
of the patient- professional relationship and employer-employee relationship into 
collaborative partnerships focused on quantifying and improving work capacity and 
enhancing the overall health-related quality of life of individuals. 
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Chapter 9 (R) 
Quantifying the Body: Body Image, Body as 
Awareness and Self-Tracking Technologies 


Arianna Boldi and Amon Rapp 


Introduction 


Among the plethora of terms used to refer to self-tracking, “Quantified Self" (QS) 
is commonly employed to describe a community of people that attempt to gain 
“self-knowledge through numbers", believing that tracking is an essential starting 
point to make a change in the direction of an “optimal self’. In Quantified Selfers’ 
perspective, precise measurements and accurate data interpretation should lead to 
better awareness and improved knowledge, informing their everyday decisions, 
shaping their future, and, eventually, their identity [1]. 

In QS rhetoric, technological devices can overcome the natural limits that people 
encounter when they seek to gain self-knowledge, like a poor sense of time, a lim- 
ited, fallible memory, and cognitive biases that negatively affect the opportunities 
for collecting relevant information to make decisions. Exact numbers collected by 
technology, instead, are powerful as they are not subject to memory distortion and, 
most importantly, "they hold secrets they can't afford to ignore, including answers 
to questions they (people) have not yet thought to ask" [2]. This belief is entangled 
with the empiricist idea that “observation” can convey a neutral, objective and clear 
comprehension of phenomena: unlike language, which is ambiguous and multiva- 
lent, data speak for themselves [3]. 

The availability of wearable devices and ubiquitous technologies recently 
boosted the popularity of self-tracking technologies even outside the strict circle of 
Quantified Selfers, reaching the broader population [4—6]. QS rhetoric then seeped 
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into the everyday use of these technologies, which are now integrated into a variety 
of practices in domains as diverse as health, sport, wellness, and safeness. 
Achieving "self"-knowledge is strongly emphasized within the QS discourse, 
but do these technologies really support the development of an integrated knowl- 
edge about the self [7]? Or do they fragment the image that we have of ourselves 
into a variety of unrelated patterns of data? In this perspective, the body and the 
representation that self-tracking technologies convey of it gain a central importance. 
Self-trackers are involved in a complex process of knowledge development, but 
this cannot be achieved without knowing the body, as knowledge is always situated 
and embodied [8]. However, self-tracking devices seem to embrace an abstract and 
scattered conception of the body, based on unrelated numbers, graphs, and depic- 
tions. This representation appears to not integrate into a coherent image that takes 
into account the body complex nature made up of perceptions, proprioceptive sen- 
sations, and self-representations. This may turn into biases and distortions of how 
we look at our bodies, worsening, rather than improving, our self-knowledge [9]. 
In order to understand the ways through which the progressive "quantification" 
introduced by self-tracking technologies is affecting the body, we need to prelimi- 
nary explore a series of theoretical constructs concerning the body, which appear to 
be addressed differently by literature pertaining to different disciplines (e.g., Human- 
Computer Interaction, psychology, sociology, neurology). This diversity may entail 
unclear definitions and theoretical overlaps that may cloud our understanding of the 
current changes produced by technology on our bodies. How are the concepts of the 
body and the self conceived? What are their relations? What kind of relationship is 
there among body schema, body image, and body awareness? These are some of the 
questions we address in the first part of the chapter. The second part, instead, illus- 
trates how individuals’ body image and awareness are affected by the usage of self- 
tracking technologies in the sports domain. It clarifies the concepts introduced 
above, by surfacing how athletes use wearable data to inform their sports practices 
and eventually develop an understanding of their body. It shows both the opportuni- 
ties and the risks introduced by the quantification of the body, by highlighting that 
self-tracking technologies may either increase the understanding of the athlete's 
body, or turn it into a series of aseptic information, which may distance the athlete 
from her body sensations and excessively "rationalize" her sports activity. This part 
builds on the empirical data collected through 20 interviews conducted with amateur 
and elite athletes, which have been previously published in a TOCHI article [9]. 


The Self and the Body 


Self-trackers are interested in achieving a better knowledge on themselves, which 
can be useful to enrich or change several aspects of their lives. At least in principle, 
this goal should be achieved by placing the body at the center of the knowledge 
development process. However, it appears that the body, in its materiality and mul- 
tifaceted nature, is clouded in self-tracking practices. 
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A core concept in critical investigations upon self-tracking is that of “digital 
double", also known as “data twin", “data double" or *datafied self’, which results 
from data assemblages [3, 10]. These investigations emphasize that, albeit we natu- 
rally have a body made up of sensations, contemporary technologies feed back a 
"screen body", an abstract object dematerialized in a variety of data points. The 
body becomes something to be observed from a distance, controlled and managed 
with the help of technology. 

Contemporary medicine is certainly the field in which technology has produced 
the most visible shift in the way bodies are treated: in the clinical practice, the 
symptoms recounted by the patients are losing their relevance, in favor of the visual 
examination mediated by technology (X-ray, RM, TAC), which is in charge of find- 
ing the "truth" about the body [11]. Likewise, self-tracking devices collect data that 
are not immediately visible and display them to the user, generating a “virtual” ver- 
sion of the body, a repository of storable and processable data [12, 13]. As an emerg- 
ing effect of the quantification of human body through biometric practices, bodies 
are turned into numbers [14]. Nonetheless, numbers are not the natural way through 
which we represent our bodies. 

People have different ways to relate to their body and, through them, they inter- 
act with the world and build their own sense of the self. Body schemata, body image 
and body awareness are theoretical constructs that point to particular ways of repre- 
senting the body. These body representations are built upon a set of sensations, 
which are the object of perception. Human beings, however, are not purely reactive 
agents and perception is not something that “simply happens to us" [15, 16] nor 
senses are passive receptors. Rather, cognitive, emotional and even cultural factors 
influence the perceptive process, even when we consider the most primary aspects 
of the body, such as the heartbeat, which are tracked and measured by self-tracking 
devices. 

This entails that sensations and body processes cannot simply translated into 
objective numbers aimed at capturing the “immediate” nature of our body. Actually, 
our relationship with our body is mediate by our representations, and there is a con- 
siderable gap between body sensations and their subjective appraisal. The goal of 
the next paragraphs is to provide a greater understanding of the ways we have to 
mediate the relationship between the body and our selves. 


Body Constructs 


The scientific literature about body representations points out six main theoretical 
constructs that operationalize the way we relate to our bodies: "body schema", 
“body image", “body awareness", "interoception", "exteroception", and “proprio- 
ception". The first three constructs concern the representations people have of their 
own body, resulting from the integration of various signals (e.g., touch, hearing, 
sight) and their processing at different levels (e.g., cognitive, sociocultural). Instead, 


the latter refer to the perceptive processes concerning the state of the body in 
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relation to endogenous and exogenous stimuli. More specifically, proprioception is 
defined as the awareness of the body position and of the movements of the body; 
exteroception refers to the perception of the body arising from exogenous stimuli; 
interoception is a multidimensional construct that concerns the perception of sensa- 
tions connected with body internal processes, like organ functioning [17-19]. 

Even though all these constructs are equally important to understand body-self 
relation, in the following we focus on the constructs concerning body representa- 
tions, as they are more tightly related with the issues arising from the use of self- 
tracking devices to monitor body parameters. 


Body Schema, Body Image, and Self-Tracking 


There is large consensus in psychological and philosophical literature over the exis- 
tence of two distinct types of body representation: body schema and body image 
[20, 21]. 

The concept of “body schema" has been first introduced by Bonnier [22] and 
further defined, by Head and Holmes [23], as a representation, mostly unconscious, 
of the body's position in space. By contrast, body image is depicted as a more con- 
scious and intentional representation of the body, or a set of beliefs about the body. 

These ways of representing the body are essential for building the self: the inter- 
nal stream of sensations, which makes a person feel the body as her own, has been 
recognized as central for developing a stable sense of identity. Contemporary neu- 
ropsychological research showed that both deficits and distortions of body schema 
and body image lead not only to a variety of deficits in bodily experiences, such as 
personal neglect, apraxia or autotopagnosia [21, 24], but also to more complex dis- 
orders, such as anorexia and bulimia nervosa [25]. Sometimes, distortions in body 
schema and body image may be intertwined in the same syndrome, without a clear 
separation between them [26]. Nevertheless, body schema and body image should 
be treated separately since they refer to different ways of representing the body, as 
we will see in the next paragraphs. 


Body Schema 


Body schema is a representation of the body's spatial properties, a constantly 
updated postural model, mainly unconscious. The first investigations on body 
schema were focused on the somatosensory capacities of our bodies and their rela- 
tion with the self [22, 23]. Later authors confirmed that the sensorimotor capacities 
of our bodies are fundamental for the construction of the self, as they shape our 
pragmatic possibilities to interact with the environment [21]. The fact that we have 
a body that moves in certain ways drives the way we perceive and act and this, in 
turn, contributes to shape our self in relation to the world. 
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The relation among sensoriality, movement, the body and, ultimately, the self has 
been further addressed by both ecological [among others, 27] and enactive or sen- 
sorimotor theories [e.g., 28]. According to the latter, sensory systems are active 
systems that function as simulators of action. Then, perception is connected with the 
ability of the brain to anticipate action by using internal models, which simulate and 
somehow predict the interaction among the body, the environment and other entities 
[15]. In other words, to perceive is, essentially, to simulate through internal models 
[29]. In this line, recent works enriched the concept of body schema by conceptual- 
izing it as an integrated internal model of the body, which represents and simulates 
the spatial properties of the body and its surroundings. 

To summarize, body schema is important for the interaction of the person with 
the environment, and having a coherent and stable body schema is essential for 
developing an integrated sense of the self situated in the world. The construct of 
body schema, however interesting, allows us to see only one side of the problem, as 
it focuses on the spatial aspects of perceiving and representing the body. It leaves 
apart representations that involve more complex factors, such as beliefs, emotions, 
and values, also including socio-cultural norms. To account for all these elements, it 
is needed to introduce a more complex construct, namely, the concept of body image. 


Body Image 


Body image points to a more conceptual representation of the body, even though a 
univocal definition of its characteristics is difficult to achieve. It appears to be con- 
nected with body schema, as the experiment of the rubber hand shows [30]. In this 
experiment, the body image acts in a top-down manner upon the body schema [20], 
making the individual believe that she feels sensations on a rubber hand. Body 
image also seems to be entangled with the evaluations people make of various char- 
acteristics of their own bodies (like shape and size), as well as the emotions associ- 
ated to those evaluations [31]. 

Body image, therefore, is a cognitive representation of the body, but is not an 
exact copy of the body as it appears from the outside (as the image that the body 
reflects in a mirror), nor of the functioning of the internal organs or the autono- 
mous nervous system [32]. Rather, body image appears to be related to the nar- 
rative aspect of the self, which concerns the stories that we tell about 
ourselves [21]. 

The close relationship between body image and the self is particularly evident in 
people with a distorted body image. Dissatisfaction with weight and body shape has 
been associated with several psychological problems: in particular, it is considered 
a predictive factor for eating disorders [31]. Moreover, researchers found correla- 
tion between Identity Problems, according to DSM IV, and body image: for exam- 
ple, Vartanian [33] emphasizes that the body defines the self and having a problematic 
body image may lead to an equally disturbed sense of the self. As there may be 
multiple representations of the body [29], individuals may have multi-faceted 
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self-definitions [34]. However, it is important that these facets are stable, coherent, 
and clear, since coherence is considered a protective factor with respect to bodily 
and identity disorders [33]. 

The complexity of body image construct is apparent if we examine how it is 
operationalized in questionnaires aimed at analyzing the body image. To assess 
body image more than 150 measures have been used [35]. Kling et al. [35] synthe- 
sized the psychometric properties of several self-report measures about body image: 
the revised Body Appreciation Scale (BAS) [36], the Body Esteem Scale for 
Adolescent and Adults (BEESA) [37], the Body Shape Questionnaire (BSQ) [38], 
the Centre for Appearance Research Valence Scale (CAR-VAL) [39], the Drive for 
Muscularity Scale (DMS) [40], the Weight and Shape Concerns Subscales of the 
Eating Disorders Examination Questionnaire (EDE-Q) [41], the Body Dissatisfaction 
subscale of the Eating Disorder Inventory-3 (EDI-3) [42] and, finally, the Appearance 
Evaluation subscale and the Body Areas Satisfaction Scale of the Multidimensional 
Body Relations Questionnaire (MBSRQ) [43]. 

This variety may depend on the multidimensionality of the construct, which led 
researches to develop different body image measures. Here, therefore, we can define 
body image as a multidimensional construct, which encompasses thoughts, atti- 
tudes, beliefs, emotions, and cultural values related to the body [44]. Body image, 
in fact, is also affected by cultural stereotypes associated e.g., to gender [45]. 


Self-Tracking 


If we consider body schema and body image as representations that mediate our 
relation with the body, we can affirm that self-tracking technologies should account 
for this mediation. By collecting and feeding back data about our bodies, they do 
not simply transform our body processes into numbers, but also affect the ways we 
represent our bodies. Likewise, the ways we look at our bodies may impact on how 
we use self-tracking technologies. What role does body image play in self-tracking 
practices? What happens when people have the availability of a large amount of 
body data, which integrate (or do not integrate) into the images they have of their 
own bodies? 

On the one hand, Edwards [45] showed that activity tracker use (i.e., Fitbit) may 
be affected by the image that users have of their bodies: dissatisfaction with body 
image does provide motivation for using a Fitbit and dissatisfied users look to 
improve their bodies in some way (N = 9; age range = 16-64; females = 5). On the 
other hand, the “schizophrenic phenomenon” can shed light on the issues that peo- 
ple are encountering when using self-tracking devices. For example, Hortensius 
et al. [46] pointed out the frustration and sense of fragility that trackers feel when 
they cannot link a measure (e.g., of their food intakes, or blood glucose levels) to 
their personal experience, or when the device prompts undesirable data (N = 28; age 
range = 40-76; females = 15). Numbers that are not coherent with the user's body 
images seem no to give her any cues for improving her self-understanding: rather, 
they can lead to a sensation of despair [47]. 
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People use narratives to constitute the self [48]. Such narratives are commonly 
built in retrospection, upon reminiscences: however, current perceptions and mental 
events play an important role in this process. The self is not something that merely 
lives in the past, through its memories, or in the future, through mental simulations: 
it exists here and now, in the hic and nunc, and it constantly changes along with our 
internal perceptions, experiences, and actions [7]. The "self", therefore, is made up 
of a multitude and mutable representations, elicited by a flow of sensations and 
bodily actions that occur in the moment. The sense of coherence we experience 
about our self is due to more stable configurations that sediment over time, like 
body image, and to the narratives we tell us about ourselves. 

Self-tracking technologies, to be effective, should then integrate into these 
aspects of the self, encouraging, rather than disrupting, a coherent image about the 
body and, consequently, a coherent narration about the self. In other words, people 
can effectively use self-tracking devices to build their identity provided that the 
"digital self" emerging from the data becomes integrated into the body representa- 
tions and self-narratives they have constructed over time. More precisely, self- 
tracking technologies can develop self-knowledge, if they are able to support people 
in generating coherent images and stories about their body and their self [3, 48]. The 
integration of the data in a coherent self guarantees a stable sense of identity and 
serves as a protective factor for mental health. 

In so doing, they should take into account the flux of mental and bodily events 
continuously affecting the "present self", especially those of which the person is 
aware. This leads to consider the notion of body awareness, which differs from both 
the concept of body schema and that of body image, albeit is strongly connected 
with both of them. 


Body Awareness and Self-Tracking 


The “body awareness" construct emerged across a wide range of health topics and 
has been described as “an innate tendency of our organism to self-organize and to 
feel the unity with oneself" [49]. It stems from the concepts of proprioception and 
interoception, but has a more nuanced meaning. 


Body Awareness 


Interoception, as we have seen, refers to the perception of sensations concerning the 
internal parts of the body, like heartbeat and respiration. Nevertheless, there is a 
distinction among the actual body-related events, their subjective perception [50] 
and the way each person evaluates her ability to accurately identify internal body 
states, which is a metacognitive skill [51]. In this perspective, body awareness is 
more than the simple focus on one's own body, as it requires recognizing the inter- 
play between body states and the cognitive appraisal of those body states [52]. 
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Therefore, body awareness may be considered as an interface between top-down 
and bottom-up information: on the one hand, there are visual, tactile, olfactory, 
gustative and proprioceptive stimuli; on the other hand, there is the cognitive- 
affective processing of those physiological perceptions [53]. 

Body awareness is a key element for affect regulation and for the sense of self 
[18, 54] and it strictly depends on mental processes, included attitudes, affects, 
beliefs, memories, and cultural imprints [54]. It seems, in fact, that those mental 
processes can modify the subjective experience of body parts and of the body in 
general. 

Pylvänäinen and Lappalainen [55] highlight that body awareness and body 
image are strictly tied together. This is evident, for example, among depressed 
patients: it has been observed that depressed individuals having dissatisfaction with 
body image also lack mindful body awareness [55]. Likewise, patients with fibro- 
myalgia overestimate their body size due to the experience of pain in certain body 
areas: “as pain increased, the patients described changes in the perception of their 
body size and its relationship with space: they felt their body becoming larger and 
as though space was shrinking” [56 , p. 2]. We may say that body image refers to a 
more stable representation of the body, which has been developed over time, 
whereas body awareness accounts for the momentary conscious stimuli that con- 
tinuously affect our bodies. Both the representations involve cognitive, emotional 
and cultural aspects. 


Self-Tracking 


Considering body awareness when we investigate self-tracking practices may allow 
better understanding the impact of self-tracking devices on the body, as well as their 
potential positive and harmful consequences. 

Sharon and Zandbergen [57] argued that self-trackers use their devices to have a 
more “active and watchful mind”, helping them be aware of body sensations, 
actions, and habits that are commonly unperceivable. Self-tracking allows people to 
sense elements concerning their internal perception, like the time of the day, or to 
acquire new capabilities, like identifying the calories and the weight of a portion of 
food just by looking at it. “In such examples, numerical data are not all the end-goal 
of tracking; they are more like an unsophisticated, intermediate stage towards more 
augmented senses.” [57 , p. 1700]. Here, self-tracking serves to raise bodily aware- 
ness, to learn to better feel the body through the data [3] and to improve the users’ 
confidence in perceiving their own body. Research confirmed that augmenting per- 
ception of body stimuli through data could improve body awareness [49, 58], and 
this could have positive impacts on people’s health [59]. 

However, paying more attention to body states by using self-tracking technolo- 
gies may not be beneficial for all the individuals and, in certain cases, it may elicit 
discomforting sensations [4, 27]. People are different, are situated in diverse con- 
texts, and have different reasons to collect data about their bodies: they may need to 
monitor very specific body aspects that may be crucial for their health, or to gain 


9 Quantifying the Body: Body Image, Body Awareness and Self-Tracking Technologies — 197 


another perspective on their bodily sensations. However, they can use self-tracking 
technologies also to reinforce some maladaptive behaviors. 

An excessive focus on the self and on the body can be linked to emotional dis- 
tress, anxiety and depression disorders, as well as eating disorders and sexual dys- 
functions in certain individuals [60]. For instance, continuous health feedback, 
prompted by tracking technology, may worsen anxiety and stress symptoms leading 
to preoccupation with one's health, especially in people with certain personality 
traits, such as neuroticism and anxiety sensitivity [4]. This entails that self-tracking 
technologies should account for the different predispositions that different individu- 
als may have, depending on their personality traits, and even on previous psycho- 
logical disorders. 

Moreover, body quantification may give an excessive emphasis to numbers and 
data to the detriment of feelings and sensations, yielding a sense of disembodiment 
[11, 61]. A virtual self, made up of disembodied data, could alienate the individual 
from herself and from the others. Berardi [62] stressed that alienation describes the 
contemporary age characterized by the impossibility of enjoying the presence of the 
other, in the form of physical presence. With the word "'derealization", he refers to 
the difficulty of the “animated body” in accessing the “animated body” of the oth- 
ers. Technologies have made remote interaction possible, that is interaction in the 
absence of the bodies [63]. Nevertheless, “in presence" social interaction is consid- 
ered fundamental for the building of the self [64]. We need others' corporality to 
grasp the nonverbal cues that tell us their attitude toward us and, finally, to under- 
stand who we are: when the bodies are substituted with data, and communication is 
replaced by sharing information, the risk is that we form a more opaque image of 
ourselves. Users, especially adolescents, who compare their “virtual body" with 
that of other users on social media platforms are more exposed to several health- 
related psychological outcomes, like anxiety, depression or sleep problems [65]. 
Technology may further worsen symptoms of people who already have trouble with 
their body image, as in the case of patients suffering from bulimia and anorexia 
nervosa using weight-loss app [66]. 

In sum, the quantification of the body operated by technology and its subsequent 
dematerialization open both opportunities for and threats to the ways we relate to 
our bodies and our selves. The double-edged consequences of self-tracking on body 
representations will be further exemplified in the next paragraphs, where we report 
on the findings collected during a qualitative study conducted with amateur and elite 
athletes about the use of self-tracking in sport. The next Section summarizes parts 
of the findings reported in Rapp and Tirabeni [9], focusing on how personal data are 
affecting the way athletes relate to their bodies. 


Self- Tracking and Sport 


We interviewed 20 athletes to investigate the impact of self-tracking technologies 
on physical activity. 
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We recruited 8 amateur athletes (A1-A8) and 12 elite athletes (EI-E12) (mean 
age = 31,7; SD = 6,5; females = 8) who have been using a self-tracking device for 
more than three months, asking them to recount their experience with it. All the 
recruited participants owned a smartphone and a wearable device aimed at captur- 
ing sports-related data. While elite athletes competed at least nationally during their 
career, amateur athletes exercised at least three times per week, spending five hours 
or more practicing. We included in our sample different sports, involving both 
endurance and non-endurance athletes. The sports addressed were cycling, swim- 
ming, triathlon, cross-country skiing, ski mountaineering, trekking, alpinism, free 
climbing, soccer, and sprint running. Almost all the athletes were educated and 
numerate. We aligned the size of the sample to other Human-Computer Interaction 
(HCI) studies with similar purposes and design, also following a data saturation 
criterion. 

The interviews were semi-structured and lasted an average of 58 minutes 
(min = 40 min.; max = 70 min). Questions were addressed to explore athlete’s atti- 
tude towards their discipline, use of personal data, and effects of use of technology 
on their sports experience. We allowed participants to explore topics not listed in the 
interview guide, and we prompted new questions when we needed to better under- 
stand their recounts. Interviews were audio recorded and transcribed. The analysis 
of the collected data followed standard open and axial coding techniques. Data were 
coded independently by two researchers who generated initial codes. Then, they 
reviewed the codes to assess their consistency. All the discrepancies were discussed 
and resolved. 

In the following we outline how self-tracking technologies affect the ways the 
athletes represent their own body. Most of the reported quotes are extracted from 
Rapp & Tirabeni [9]. 


Findings 


Self-tracking devices, at first glance, appear to have a positive impact on body 
awareness, especially for amateur athletes. Al, for instance, reports that such 
devices provide him with “an awareness that you couldn’t have before”. Amateurs 
agree that trackers can support the athlete in developing a greater awareness of her 
body, by prompting fixed measures to which compare those signals that are tied to 
a specific level of heart rate. Being in a certain hear rate zone, in fact, is a primary 
goal for athletes who want to achieve a certain standard of performance: “if you’re 
within the zone and you know how you feel, then you try to memorize it, and then 
when you do a race or a workout, and you’re without the heartbeat [tracker], you try 
to understand in which zone you are, if you’re in a medium that you can manage for 
the whole race", says EA [9]. 
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The device not only can make the athlete directly aware of internal body pro- 
cesses that she is not able to identify by herself (i.e., the heart rate); but also can 
support the athlete in learning how to “read” her body, in order to detect such hidden 
processes. When the athlete feels certain body signals, she may not be able to retrace 
them to a specific heart rate zone. This is due to their high variance: they may differ 
depending on contextual factors, like the weather or the athlete's physical condition. 
The device allows the athlete to progressively bring those signals that are meaning- 
ful for her sports performance back to certain heart rate zones, thus "teaching" her 
how to become more aware of the internal processes of her own body. 

The increased body awareness that self-tracking technologies may produce, 
however, is not exempt from side effects. In fact, the device, rather than being used 
as a tool for learning how to listen to the body, may become essential for the ath- 
lete's sports practice. The elite athletes emphasize that self-tracking devices may 
undermine the athlete's confidence in what they call “sensations”, in favor of a 
complete reliance on the data provided by the device. Such sensations refer to the 
body and go beyond the signals of being in a certain heart rate zone. Actually, they 
point to fine-grained information about the body that allows the elite to tune her 
performance on the basis of the continuous changing context. It is a superior form 
of body awareness that elite athletes develop over the years, by carefully listening to 
their bodies: “To use sensation means that I search some reference points in my 
body, the rhythm of the hair on the shoulders, how the foot hits the ground, if it's 
heavy, or more round... [...] It's even the sensation that I have at that moment. Some 
days when I don't want to push forward at all... and then I precisely hear the exer- 
tion of the legs, the sensation of being more or less light" [9]. These body sensations 
are used to tune their sports performance, regulate their rhythms, understand their 
level of fatigue, and recognize when they are reaching their body limits. 

Awareness of sensations, however, can be jeopardized by an excessive use of 
self-tracking devices, which, in turn, can worsen the sports performance. E11 high- 
lights that “it happens to see athletes, non-professional athletes, athletes of the next 
generation... you tell them ‘run slow for an hour’ and they're not capable of running 
slow because they don't have a reference, they don't have the watch [the tracker] 
that can tell them that they're running slow, they can't manage themselves" [9]. 
Elite athletes use their device simply as a commodity, rather relying on their “supe- 
rior form" of body awareness during races, when the technology is actually left apart. 

The tracker, in fact, may also give information contrasting their current body 
representation, and this may produce anxiety and worries during important events. 
E10 says that “So many times you feel good and you push forward, and maybe the 
heartbeat goes beyond the rate that you think you should keep, and maybe by look- 
ing at the watch you get frightened thinking that the rhythm that you're keeping is 
wrong, when maybe your body is actually adapting itself [...], you're managing 
everything all right, even if it's a little harder than what you had set in advance" [9]. 
The discrepancy between the representation of the body prompted by the device and 
the representation owned by the athlete may thus be perceived as disturbing and 
counterproductive for the athlete's goals. 
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This situation, however, is slowly changing due to technological advancements. 
More fine-grained instruments are progressively allowing the measurement of “sensa- 
tions” that were previously identifiable only by the athlete. In cycling, for instance, the 
power meter allows to capture the cyclist’s legwork. “It has been the real revolution of 
both workouts and competitions, for now only available for bikes [...] If you keep 395 
you'll blow away at the last kilometer. If you see others that maybe begin the rise 
stronger than you, you don't care, you look at your instrument, you keep that power, 
because you know that you can keep it’, says E8 [9]. Nonetheless, in the elites’ eyes, 
"these advancements" are seen as a worsening of the overall sports experience. 
Despite the undeniable positive impacts of devices such as the power meter on the 
sports performance, such devices are slowly affecting the athlete's body image, shift- 
ing it from a “living body" to a “mechanical body”. E7, for instance, emphasizes that 
"the watch is a machine and measures your activity as if you were a machine, but the 
human body... there is a mental part and other mechanisms that the watch can't com- 
pute" [9]. This points out that the complex nature of the body can hardly be turned into 
numbers without producing impacts on the body image. 


Discussion 


Findings of this study highlight that the body is so variable in its reactions to both 
endogenous (e.g. stress, fatigue) and exogenous (environmental) factors, that 
parameters, like the heart rate, which technology aims to capture and turn into uni- 
vocal numbers, can hardly account for it. The body is made up of meaningful *sen- 
sations", and the body awareness that elite athletes develop is addressed to detect 
the richness of such sensations. The increased awareness about the heart rate (and 
other “objective parameters" pertaining to the body) produced by self-tracking tech- 
nologies, therefore, may cloud the athlete's opportunities for recognizing body sen- 
sations. Actually, it may decrease the awareness of the whole body in its multifaceted 
variability. In other words, the greater awareness of body parameters induced by the 
data may mislead the athlete's “superior” body awareness, which is considered by 
the elites more reliable and fruitful when there is a lot at stake. Furthermore, it may 
prevent the development of such ability in the amateur athletes. A subsequent study 
substantially confirmed the insights coming from this research [67]. 

However, the development of more "precise" devices, capable of directly tying 
the body measures to the sports outcomes, seems to be progressively changing this 
landscape. The power meter, widely employed in professional cycling, anticipates a 
future when self-tracking devices could provide an efficient substitute, in terms of 
their instrumental value, of the sensations that are currently leading the elite's con- 
duct in a large variety of sports. This, however, is seen as an impoverishment of the 
sports experience also by those who are using this kind of device. Moreover, elite 
athletes emphasize that these instruments might change their body image, trans- 
forming it in something that is shaped by the data collected by the device, to which 
the real body would need to adapt. This sort of *mechanical body" would regulate 
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the rhythms and dynamics of the real body, constraining it to respond to the incom- 
ing data in a continuous and unavoidable feedback loop. 

The concerns about the body arisen by our participants parallel those empha- 
sized by authors who are starting to outline a critical discourse towards the assump- 
tions embedded in QS culture. Lupton [11], for instance, highlights that trackers 
appear to extend the capacities of the body by supplying data that can then be used 
to display the body’s limits and capabilities and allow users to employ these data to 
work on themselves. However, these technologies conceptualize the body not as a 
sensing body through which one can gain self-knowledge, but as a data generating 
device that has to be coupled with technology in order to be known [13]. In this 
perspective, the repository of the body knowledge shifts from the individual to the 
device. This also entails the individual’s subservience to technology, since these 
“data-doubles” feed back information to the user in ways that are intended to 
encourage the user’s body to act in certain ways. 

What seems relevant, therefore, is that the benefits on body awareness that self- 
tracking instruments are bringing in the sports domain may be blurred not only by 
the reduction of the athlete’s ability in becoming aware of her body sensations, but 
also by the athlete’s loss of control over her own body, induced by an externalized 
body image that is imposed by the device. 


Beyond the Athletes 


As we have already noticed, the availability of a variety of commercial wearable 
devices that automatically collect personal body data has boosted the popularity of 
self-tracking outside the circle of specific populations, like quantified selfers, ath- 
letes, and people with a health condition, reaching the general public [68]. A variety 
of smartwatches, activity trackers, and smart clothes [69-71] are now available on 
the market, promising to collect data on body aspects as diverse as blood pressure, 
body movements, and respiration. 

Nonetheless, we are currently far away from the possibility of providing such 
"general users" with complete and reliable representations of their body, which 
could be used to increase their body awareness and feed back an insightful image of 
their body. Trackers still exclusively focus on “objective” parameters and the repre- 
sentations that they give of the body is rarely meaningful for people that are not used 
to manage a large amount of quantitative data [72]. 

What kind of data and/or design techniques do we need for supporting users in 
increasing their body awareness and developing their body image, even helping 
them improve the perception they have of themselves? If we look at, for instance, 
Body Appreciation Scale 2 [36], which is a positive body image measure assessing 
individuals’ acceptance of, favorable opinions toward, and respect for their bodies, 
we see the multidimensionality of body image construct and, consequently, the 
complexity of capturing and feeding back an image of the user's body that could 
really help her ameliorate the image of her body. 
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Scale items span from body acceptance and love (e.g., “I feel love for my body”, 
“I feel good about my body"). inner positivity influencing outer demeanor (e.g., 
“My behavior reveals my positive attitude toward my body; for example, I walk 
holding my head high and smiling"), to appreciating the functionality of the body 
(e.g., “I feel that my body has at least some good qualities"), taking care of the body 
via healthy behaviors (e.g., "I respect my body,” “I am attentive to my body's 
needs”), and internalization of media appearance ideals (“I feel like I am beautiful 
even if I am different from media images of attractive people (e.g., models, actresses/ 
actors"). In order to take into account all these aspects of the body image construct, 
it is not possible to simply rely on the functionalities of current trackers, but we need 
to envision novel ways of collecting and visualizing data. 

For instance, self-reporting appears essential to grasp the subjective meanings 
that people ascribe to their body, and thus capture body acceptance and love, as well 
as appreciation and taking care of the body. Future research, therefore, needs to 
explore novel ways for eliciting the self-reporting of body data, which may be bur- 
densome by requiring a high degree of compliance [73]. Self-reporting may be 
complemented by content analysis of social media, especially with reference to the 
goal of understanding how media appearance ideals may affect the user’s body 
image; or automated tracking focusing on “specific aspects” of body data, like those 
related to *body-harm" (e.g., lack of sleep, bad food, and lack of exercise), which 
could help to infer how people take care of their own body; or those connected with 
posture and face expressions/emotions (maybe collected when the user is looking at 
herself in front of a mirror), which may work toward a better understanding of the 
inner positivity toward the body. 

Furthermore, finding novel ways for representing body data becomes essential if 
we want to give a meaningful body image of the user, also pushing her toward a 
more positive representation of her body. Adopting concrete representations of the 
body data collected by a wearable may support the projection into and the develop- 
ment of an emotional connection with them [74], possibly promoting the develop- 
ment of a more positive body image. Rapp et al. [6], for instance, proposed a 
visualization of the user's body merged with the personal data collected by a variety 
of self-tracking devices, as if the user were looking into a mirror. The visualization 
does not allow for a precise quantification of the user's data, but conveys a general 
impression about the user's body, engaging her, at the same time, in an immersive 
interaction. 

All these lines of research, which could work toward making self-tracking data 
closer to people's needs, by providing them with more meaningful body representa- 
tions also pushing them toward more positivity, are still in their infancy. Their 
exploration would lead to design novel Quality of Life Technologies (QoLT), which 
have been defined as any technologies for assessment or improvement of the indi- 
vidual's QoL [75]. In fact, allowing people to construct a more positive image of 
their own body, as well as to develop a greater body awareness, could ultimately 
increase their overall self-knowledge providing benefits to many different aspects of 
their everyday life. However, much more research is needed to find insightful depic- 
tions of body data and novel ways to unobtrusively collect them. 
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Conclusion 


In this chapter we have highlighted the potential impacts of self-tracking technolo- 
gies on the body. In order to assess such impacts, we stressed that we need to con- 
sider the ways we naturally relate to our bodies, namely through different body 
representations. On the basis of the examination of three theoretical constructs that 
refer to the different ways through which we represent our own body, we outlined 
how technology is affecting our body conceptions, particularly highlighting the 
potential negative outcomes that may stem from the usage of body data. In this line, 
examples coming from a study in the sports domain pointed out the main risks for 
the athlete’s body opened up by tracking technology. 

However, all the issues we pointed out may be counterbalanced by the great 
opportunities that self-tracking instruments seem to open. Esmonde [68] highlights 
that the boundaries of the body are not defined by the skin, as they extend to the 
outside world, the environment and technology. In this perspective, self-tracking 
devices could have the potentialities to expand the limits of our body and, eventu- 
ally, of our self. To achieve this goal, however, researchers and practitioners should 
start rethinking not only the way technology is designed, but also the theoretical 
frame in which it is inserted, in order to account for the complex modalities through 
which we relate to our bodies and our selves. 
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you May Think: Emotional Assessment *on 

the Go’ Via TapSense 


Surjya Ghosh, Johanna Lóchner, Bivas Mitra, and Pradipta De 


Introduction 


Emotions have an enormous impact on our momentary performance, health, and 
way of relating to others, hence on the quality of a persons’ life. In particular, the 
experience of unpleasant (or pleasant) emotions is directly related to an individual's 
well-being. Emotions are influenced by subjective experiences and memories and 
the context the individual is in, and it seems almost impossible to measure this phe- 
nomenon objectively, reliably, and validly. Indeed, capturing human emotional 
states has been a challenging task for researchers for decades, leading to numerous 
theories about emotions, moods, and feelings. Specifically, psychometrics focuses 
on the theory and techniques of psychological measurements, including the QoL 
measurements. The emerging field of affective computing promises to overcome 
some methodological difficulties that lead to limitations in traditional methods of 
psychometrics. Affective computing is the study of technologies that can quantita- 
tively measure human emotion from different clues. It is based on the hypothesis 
that an individual's digital footprint is highly correlated with their perceptions, feel- 
ings, and resulting behaviors and that extracting and analyzing this data collected 
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over time can prove that “your smartphone knows you better than you may think.” 
In addition to the fact that people use their digital devices extensively, being the first 
and last thing used during a normal day [1 ]—this statement becomes even more valid. 

In this chapter, we discuss the current development within the affective comput- 
ing area while focusing on assessing emotions via personal technologies. Firstly, we 
define emotions as a complex interplay of different components (sensory, cognitive, 
physiological, expressive, motivational) over time. Several emotion theories have 
been developed in the past years, along with emotions being classified into three 
dimensions: valence, arousal, and dominance [2]. Among these emotion theories, 
the Component Process Model [3] stands out, revealing the emotional process that 
leads to an individual’s perception and processing of negative and positive life expe- 
riences. As an extension of this model, the Emotional Competence Model [4] 
hypothesizes that mental well-being and adverse psychopathology (e.g., anxiety, 
depression) greatly depends on a well-functioning emotional process. This depends 
on the individual’s experienced emotional response, perception of the situation, 
adequate appraisal, and emotional regulation. Hence, emotional competence plays 
a key role in maintaining a person’s quality of life. Furthermore, the knowledge and 
perception of emotions are basic abilities that may elicit more adaptive emotion 
regulation strategies (like an acceptance of an uncontrollable stressor). 

To capture individuals’ emotions (e.g., for clinical or research reasons), mostly 
self-reports for negative and positive emotions are conducted using psychometri- 
cally validated questionnaires of concepts as, e.g., stress, depression, and well-being 
with a recall period of days to months. These assessment instruments face the prob- 
lem that (i) self-reports are rather subjective and often biased by, e.g., time and 
motivation to fill out often many questions and (ii) they are influenced to a great deal 
by the current psychological state interfering with the recall of someone’s mood 
days to weeks ago. To overcome these issues, new assessment methods arose in the 
past years using personal digital technologies. For example, the Experience 
Sampling Method (ESM), also known as Ecological Momentary Assessment 
(EMA) [5-7], is increasingly used in psychology to trigger self-reports for emotions 
and behaviors momentarily, i.e., as closely as possible to the subject’s daily life 
experiences, periodically (randomly or at fixed intervals) or in an event-driven fash- 
ion [6, 8]. 

Traditionally, different modalities such as facial expression [9-13], speech pros- 
ody [14-20], physiological signals like ECG, EEG, HR, GSR blood, brain, posture, 
[21-27] are explored for emotion assessment. Additionally, other sources can be 
used to extract emotions using smartphones and internet usage, which is discussed 
in this chapter. To determine the emotion states, often these affect-aware systems 
deploy a machine learning model. Therefore, conventional machine learning mod- 
els like Support Vector Machine (SVM), Random Forest, Bayesian approaches were 
used in affective computing [28, 29]. In these approaches, first, a set of features 
(which can distinguish one emotion from another) were extracted manually from, 
e.g., the physiological signals. These features were correlated with the emotion 
ground truth labels (self-reports) to construct the emotion inference model. With the 
latest advances in the Deep Neural Network models, the conventional approaches 
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were replaced by state-of-the-art AI models such as Convolutional Neural Network 
(CNN), Long Short Term Memory (LSTM), Recurrent Neural Network (RNN) [28, 
29]. The advances in this field have helped to overcome the manual feature engi- 
neering effort and helped obtain very high classification performance in the affect 
classification. The advances in affective computing have led to many affect-aware 
applications such as emotion-aware music player, affective tutor, mood monitor, 
which influence the quality of life [30-34]. The key working principle of such 
emotion-aware applications is to collect physiological and behavioral data from dif- 
ferent modalities and to train a machine learning model for emotion inference. 

Given the current accelerating scale of developments in personal technologies, 
new assessment techniques for emotions emerged. We present the design and devel- 
opment of a smartphone keyboard interaction-based emotion assessment applica- 
tion. Specifically, among different types of interactions performed in smartphones, 
keyboard interactions are highly interesting. They represent the input/output inter- 
action between the user and the phone for, e.g., information, communication, or 
entertainment [35]. Additionally to the interaction itself, the interactive content may 
be of high interest. The research shows that not only individuals often express 
momentary emotions on social networks’ platforms [36, 37], but also a person’s 
language was found to reveal his/her momentarily psychological state [38]. 

In our research, we focus on the smartphone interaction itself. Hence, we have 
designed and implemented an Android application TapSense, which can unobtru- 
sively log users’ typing patterns (without actual content) and trigger self-reports for 
four types of emotion (happy, sad, stressed, relaxed) leveraging the ESM method. 
Different typing features like typing speed and error rate are extracted from the typ- 
ing data and correlated with the emotion self-reports to develop a personalized emo- 
tion assessment model. However, as the conventional ESM-driven self-report 
collection for model construction is labor-intensive and fatigue-inducing, we also 
investigate how the self-report collection approach can be further optimized for suit- 
able probing moments and reduced probing rate. So, we also have developed an 
adaptive 2-phase ESM schedule (integrated into TapSense), which balances the 
probing rate and self-report collection time and probes the individual at the oppor- 
tune moments. The first phase balances between probing rate and self-report collec- 
tion time and trains an ‘inopportune moment assessment’ model. The second phase 
operationalizes such a model so that no triggering is done at an inopportune moment. 
We investigate the implications of this ESM design on the emotion classification 
performance. 

We evaluate the proposed approach in a 3-week ‘in-the-wild study involving 22 
participants. Our first important result demonstrates that using smartphone key- 
board interaction; we can determine the emotion states (happy, sad, stressed, 
relaxed) with an average AUCROC of 78%. The next major result shows the perfor- 
mance of the proposed ESM approach. It demonstrates that the proposed 2-phase 
ESM schedule (a) assesses inopportune moments with an average accuracy 
(AUCROC) of 89% (b) reduces the probing frequency by 64% (c) while enabling 
the collection of the self-reports in a more timely manner, with a reduction of 9% on 
average in elapsed time between self-report sampling and event occurrence. The 
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proposed design also helps to improve self-report response quality by (a) improving 
valid response rate to 96% and (b) yielding a maximum improvement of 24% in 
emotion classification accuracy (AUCROC) over the traditional ESM schedules. 

The chapter is organized as follows. Firstly, in Sect. 2, we discuss the definitions 
and the nature of emotions and their importance for the daily life experience, which 
influences an individual’s quality of life. In Sect. 3, we present traditional psycho- 
metric assessment instruments for emotions and different assessment methods 
leveraging affective computing developments and beyond. The background, study 
design, and empirical evaluation of TapSense—a smartphone-based approach for 
assessing emotions are presented in Sect. 4. We discuss and conclude the chapter 
findings in Sect. 5. 


Background and Related Work 


This section presents the definition and models covering the concept of emotions 
(2.1), their importance for the quality of life (2.2). It attempts to capture emotions 
via traditional (2.3) and more novel, data-driven approaches (2.3). 


Definition of Relevant Domain Concepts 


Emotions, moods, and psychological states play a key role in our personal, profes- 
sional, and social life. Also, interpersonal relationships, professional success, and 
mental well-being depend greatly on how we cope with stressful events and navi- 
gate adverse emotional experiences. Often the terms: emotions, effects, feelings, 
and moods are confused in language usage. Feelings most commonly involve invari- 
ably a direct response of the autonomic nervous system (ANS) involving organ 
functions (e.g., change in respiration pattern, adrenaline rush). At the same time, an 
umbrella term refers to all basic senses of feelings, ranging from unpleasant to 
pleasant (valence) and from excited to calm (arousal). Moods differ from feelings, 
emotions, and affects in that they are experienced as extended in time (c.f., mood 
stability by Peters, 1984) but are also subject to certain situational fluctuations [39]. 
Very similar is the psychological concept of state, referring to a person’s mental 
state at a certain point in time, introduced by Cattell and Scheier [40] as a counter- 
part to the concept of timely persisting (personality, motivational, cognitive) traits. 
In contrast, emotions are a much more complex mental construct consisting of sev- 
eral components as the physiological response and lasting from minutes to hours. 
Research on emotions is usually based on the central evolutionary importance of 
emotions for human survival. It defines emotions as “a genetic and acquired moti- 
vational predisposition to respond experientially, physically and behaviorally to 
certain internal a02nd external variables" [41]. In the context of survival, emotions 
imply complex communication patterns and information [42-44], as the feedback 
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of an individual's inner state on different levels enables a biological adaptation to 
the physical and psychosocial environment. Therefore, emotions are further viewed 
as complex, genetically anchored behavioral chains that contribute to an individu- 
al's homeostasis through various feedback loops [45]. Since the research of Ekman 
[46], it has become known that elementary emotions such as fear, joy, or sadness 
show themselves independently of the respective culture. These basic emotions are 
closely coupled to simultaneously occurring neuronal processes. However, how 
people communicate and express visible parameters, such as facial expression, are 
influenced by the values, roles, and socialization practices that vary across cultures 
[47, 48], age, and gender [49]. 

Emotions can be divided categorically into primary, secondary, and combined 
forms: primary emotions are fundamental, while secondary emotions are emotions 
about emotions (such as guilt over gloating). Ekman distinguishes six basic emo- 
tions: happiness, sadness, anger, fear, surprise, and disgust [46]. In contrast, Izard 
[50] speaks of ten fundamental emotions (1) interest/excitement, (2) pleasure/joy, 
(3) surprise/fright, (4) sorrow/pain, (5) anger/rage, (6) disgust/repugnance, (7) dis- 
dain/contempt, (8) fear/contempt, (9) shame/shyness/humiliation, and (10) guilt/ 
repentance. Another way to categorize emotions relates to their highly variable mul- 
tidimensional nature: emotions can be categorized into the dimensions as positive or 
negative (polarity/valence), strong or weak (intensity/arousal), easy or hard to 
arouse (reactivity), and based on the situation they occur (idiosyncratic vs. universal 
situation) [51]. Following partly those dimensions, Russel's Circumplex model is 
the most commonly used emotion model to capture emotion on a continuous scale 
[52]. It represents every emotion as a tuple of valence and arousal. There exist also 
a valence, arousal, and dominance model (more commonly known as the VAD 
model), which captures every emotion as a set of triplets (valence, arousal, domi- 
nance) in a continuous scale [2] (see Fig. 10.1). 

Following the earlier mentioned, there exists an empirically validated theoretical 
process model of emotion—the Component Process Model (CPM) [54] and the 
Emotional Competence Process Model (ECP), further implying the adaptive and 
maladaptive emotional functioning [4]. In the CPM, emotions are identified within 
the overall process in which low-level cognitive appraisals, particularly relevance 
processing, trigger physical reactions, behaviors, and subjective feelings (Fig. 10.2). 
As a foundation, CPM provides a differentiated theory of the various dimensions of 
emotions. Emotions are interpreted here as synchronizing several components that 
interact over time during a defined process regarding the emergence, appraisal, 
awareness, regulation, and knowledge of emotions. For example, an emotional situ- 
ation emerges due to a specific trigger (e.g., a job interview), evoking in the appraisal 
(“I hope I perform well—the interviewer does not look friendly at all") which 
results in a certain emotional reaction (e.g., adrenaline release, sweating, nervous- 
ness, tension). At the moment, a person is aware of this condition and the fact that 
they can regulate their condition (e.g., by taking a deep breath). They can categorize 
their emotional reaction (and those of others in return). 

The emotional response components cover sensory, cognitive, physiological, 
motivational, and expressive components [55]. Initially, the sensory component 
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Fig. 10.1 The VAD (Valence-Arousal-Dominance) model spanned across the six basic emo- 
tions [53] 


enables a subject to recognize an emotional event through the senses (e.g., see, feel). 
Through the cognitive component, the individual can identify possible relation- 
ships between itself and the event based on its subjective experiences. The individ- 
ual then makes a subjective evaluation of the perception of the event (appraisal). 
This goes along with the two-factor theory of emotions. As early as the 1960s, 
Schachter and Singer pointed to the cognitive evaluation of a physical response as 
key to the subsequent emotional sensation [56]. A subject can react to the same 
event with a different evaluation—depending on his personal world view, value 
system, and current physiological state. It is resulting in different physical responses 
to feelings. Depending on the subjective evaluation outcome, the individual reacts 
by releasing certain neurotransmitters and hormones, thus changing its physiologi- 
cal state (physical component). This altered state corresponds to the experience of 
an emotion. According to Lazarus’ appraisal theory [57], the emotional experience 
first arises through cognitive evaluation and interpreting an emotional stimulus as 
manageable or not. The motivational component follows the event’s evaluation 
and is modulated by the current physiological (or emotional) state. The motivation 
to a certain action of a person is oriented to an actual-target comparison and the 
prediction of the effect of conceivable actions. For example, the emotion of anger 
can result in both the motivation for an attack action (e.g., in the case of a suppos- 
edly inferior opponent) and the motivation for a flight action (e.g., in the case of a 
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Fig. 10.2 Comprehensive illustration of the Component Process Model of Emotion (adapted from 
Scherer 2001) [58] 


supposedly superior opponent). The expressive component refers to the way emo- 
tion is expressed. This primarily concerns nonverbal behavior, such as facial expres- 
sions, speech, and gestures. 


Importance of Feelings Moods, Psychological States 


From an evolutionary perspective, emotions play a very important role in motiva- 
tion, behavior, and attention: they make us act, direct our attention to certain stimuli 
that might have pleasant or unpleasant consequences for us, and give us signals so 
that we adjust our behavior to obtain or avoid those consequences. In a modern 
context, it is not only about our survival and related incentives but also about sec- 
ondary incentives, such as money, status, entertainment, or others. Moreover, emo- 
tions regulate the intensity and duration of different behaviors and cause the learning 
of those behaviors that were successful under certain conditions (e.g., joy has a 
pleasant effect on us and motivates us to repeat the behavior) and mark in memory 
(e.g., via disgust, anger) those that led to failure. In the same way, emotions function 
in regulating our social interactions, being reinforced by a pleasant (or non-pleasant) 
effect. This leads to the formation of bonds or rivalries, which provide us with ori- 
entation in the social structure. Some distinct emotions may have even a more 
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Table 10.1 Example Functions of Emotions 


Emotion | Function 


Fear Protective function: Avoidance of danger, injury, creating distance to source of danger 


Anger | Destruction function: Overcoming something that stands in the way of need 
satisfaction (e.g. nutrition) 


Pleasure | Intake function: Receiving positive stimulus from the external world that supports or 
nourishes the individual 


Disgust | Rejection function: Excrete harmful substance 


Surprise | Orientation function: Increased attention to take in information about a new stimulus 


specific function, as shown in Table 10.1 [59, 60]. It is part of our daily lives to be 
confronted with the experience of such emotions and to respond to them. How 
people deal with different emotional events varies widely. The distinction between 
“positive” and “negative” emotions is questionable due to emotions’ functionality. 

As depicted in the ECP, there is a great body of literature showing that both posi- 
tive and negative emotions greatly impact our well-being and mental health (see 
meta-analysis [61]). It has been demonstrated, for example, that people suffering 
from depression have difficulties in identifying (Rude & McCarthy, 2003), bearing 
and accepting their emotions [62, 63]. In numerous disorders, the presence of unde- 
sirable affective states (such as anxiety or depressed mood) in an inappropriate 
intensity or duration is among a diagnostic criterion of the disorder (e.g., in anxiety 
disorders or depression). Also, a whole range of cognitive and behavioral symptoms 
of mental disorders can be understood as dysfunctional attempts to avoid or termi- 
nate such undesirable states. Examples include alcohol or substance abuse, self- 
injurious behavior, or eating attacks. 

Given the nature of variety in the emotional response and experience regarding 
intensity, duration, personality, and situational aspects, it is poorly defined when a 
certain emotional phenomenon can be considered inappropriate, abnormal, or even 
psychopathological [64]. This poses a challenge to psychometricians, researchers, 
and clinical diagnosticians to make the most valid (clinical) judgment or classifica- 
tion and eventually initiate appropriate and effective treatment. 


Assessment of Emotions 


Traditional assessment of emotions is based on self-reports via questionnaire and 
interview (3.1).! For many years now, other approaches aiming to capture a person's 
emotions more objectively and independently than subjective self-reports were 
developed. In the following, assessment methods deriving from physiology-based 


! Of note, not all psychometric approaches for assessing the whole emotional process (or parts of 
it) can be displayed here. Further research attempts to improve questionnaires’ reliability, like eye- 
tracking experiments (while filling in a questionnaire), appraisal biases, and knowledge of 
emotions. 
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emotion assessment such as blood and brain (3.2), expression-based emotion assess- 
ment such as facial expression, speech, and posture (3.3) are discussed. Moreover, 
digital footprints of emotions can be approached via social network platforms and 
smartphone data collection (3.4), including momentary ecological assessment. In 
the final part of this section, we compare the advantages and disadvantages of the 
presented assessment methods (3.5). 


Self-Report-Based Methods for Emotion Assessment 


In the traditional psychometric approaches, emotions are usually measured with 
self-report questionnaires, leveraging time- and cost-efficient instruments and 
enabling access to the cognitive component of someone's emotions. It is noteworthy 
that those instruments aim to assess single emotions and approach “concepts” as 
stress, depression, or well-being, indicating well-functioning (or dysfunctional) 
emotional processes over time. Questionnaires differ on how they are evaluated 
regarding their standardized psychometric properties as test objectivity, reliability, 
and validity [65]. Usually, factor analyses are conducted to evaluate the underlying 
constructs’ factor structure in the designed questionnaire. Standardization is usually 
carried out using large and representative samples, which allow the classification of 
individual test results compared to the norm sample. Ideally, t-values or percentiles 
are defined for this purpose based on extensive psychometric studies. For the dis- 
tinction between pathological and healthy reactions and clinical diagnosis, clinical 
samples are also collected. 

To capture the emotion self-reports in the general population, often different 
scales are used, guided by the above-mentioned emotion models. In Table 10.2, an 
overview of some well-established questionnaires, including the item number and 
recall period, is given. For example, the Self-Assessment Manikin (SAM) is a non- 
verbal pictorial assessment technique that directly measures the pleasure, arousal, 
and dominance associated with a person's affective reaction [77]. Similarly, there is 
an Affect Balance Scale (ABS), based on a model that posits the existence of two 
independent conceptual dimensions—positive effect (PAS) and negative affect 
(NAS)—each related to overall psychological well-being by an independent set of 
variables [68]. The more widely accepted PANAS scale is a 10-item mood scale that 
comprises the Positive and Negative Affect Schedule (PANAS) [67]. Additionally, a 
flourishing scale determines psychological flourishing and feelings [92] in relevant 
areas such as purpose in life, relationships, self-esteem, feelings of competence, and 
optimism. 

To make a clinical diagnosis of an affective disorder, clinical questionnaires are 
based on the clinical diagnostic criteria: symptom description, duration, intensity, 
distress, and psychosocial consequences based on the classification criteria of 
DSM-V [93] and ICD-10 [94]. However, for a valid diagnosis, a structured clinical 
interview is the gold standard (e.g., SKID [95]. Nevertheless, clinical questionnaires 
are frequently used to validate the diagnosis and evaluation of the treatment process 
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Table 10.2 Overview of example, well-established questionnaires assessing emotions 


Number Recall 
Construct Scale Name of Items  |Period | Reference 
Affect The affect balance scale 10 Past few | Bradburn (1969) [66] 
(ABS) weeks 
Affect The positive and negative | 20 n/a Watson et al. (1988) 
affect scale (PANAS) [67] 
Affect Affect balance scale 10 Past few | Moriwaki (1974) [68] 
weeks 
Anxiety Manifest anxiety scale 50/28 n/a Taylor (1953) [69] 
Anxiety Self-rating anxiety scale | 20 Overall | Zung (1965) [70] 
Anxiety Beck anxiety inventory 21 n/a Beck et al., (1988) [71] 
Anxiety, The depression anxiety 21 Past Antony, Bieling, Cox, 
depression stress scales week Enns, & Swinson 
(1998) [72] 
Anxiety State-trait anxiety 40 Current | Spielberger (1989) [73] 
inventory 
Anxiety Generalized anxiety 7 Past Spitzer et al., (2006) 
Disorder-7 (GAD-7) 2 weeks |[74] 
Anxiety The Hamilton anxiety 14 n/a Hamilton, (1959) [75] 
rating scale 
Coherence Sense of coherence scale | 29 Overall | Antonosky, (1993) [76] 
Emotions Self-assessment manikin |3 Current | Bradley & Lang (1994) 
[77] 
Depression Patient health 9 Past Kroenke, Spitzer, & 
questionnaire (PHQ-9) 2 weeks | Williams, 2001 [78] 
Depression The hospital anxiety and | 14 Past Zigmond & Snaith, 
depression scale week (1983) [79] 
Depression Beck depression inventory | 21 Past Beck, Steer, & Brown 
2 weeks | (1996) [80] 
Depression Zung self-rating 20 Current | Zung, (1965) [70] 
depression scale 
Depression Center for Epidemiologic |20 Past Lewinsohn et al., 
Studies Depression Scale week (1997) [81] 
(CES-D) 
Depression Carroll rating scale (CRS) | 52 n/a Carroll, Feinberg, 
for depression Smouse, Rawson, & 
Greden, (1981) [82] 
Emotional Geneva emotional 110 Current | Schlegel & Mortillaro 
competence competence test (GECo) (2019) [83] 
Emotion Geneva emotion 83/42 Current | Schlegel, Grandjean, 
recognition recognition test & Scherer (2014) [84], 
(GERT/-S) [85] 
Psychopathology | Symptom-checklist 90 Past Hamilton, (1959) [75] 
SCL-20-R week 
Rumination Ruminative response scale | 22 n/a Treynor, Gonzalez, & 


Nolen-Hoeksema, 


(2003) [86] 
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Table 10.2 (continued) 


Number | Recall 

Construct Scale Name of Items  |Period | Reference 

Stress Perceived stress scale 14 Past Cohen, Kamarck, 
perceived stress test (PSS) month Mermelstein, (1983) 

[87] 

Well-being General health 60 Past few | Goldberg & Hillier 
questionnaire weeks | (1979) [88] 

Well-being Warwick-Edinburgh 14 | Past | Stewart-Brown, et al. 
mental Well-being 2weeks | (2011) [89] 
scale—WEMWBS 

Well-being EuroQol instrument 15 Current | Herdman et al.,(2011) 
EQ-5D-3L [90] 

Worrying Penn State worry 16 n/a Hopko et al., (2003) 
questionnaire—(PSWQ) [91] 


and follow-up. For example, the Beck’s Depression Inventory (BDI-II, [80]) is com- 
monly used in research to diagnose depression. The BDI-II contains 30 items refer- 
ring to depressive symptoms someone may have experienced the past two weeks 
with different intensity. Assessing depressive symptoms in nine items, the PHQ-9 
corresponds to the depression module of the Patient Health Questionnaire (PHQ, 
[78]). Unlike many other depression questionnaires, the PHQ-9 captures one of the 
nine DSM-IV criteria for diagnosing “major depression" with each question. Again, 
due to depression classification criteria, the recall period is two weeks. The 
Depression Anxiety Stress Scale (DASS, [72]) covers 42 Items on those three 
related emotional states in the last week by not focusing on one specific affective 
disorder. In contrast to assessing mental illness, there are fewer instruments for 
assessing mental well-being. For example, the Warwick-Edinburgh Mental Well- 
Being Scale (WEMWBS) aims to assess positive feelings an individual may have 
experienced to a certain extent in the past two weeks. 

Overall, questionnaires to assess emotions (for clinical use) have the advantage 
of not requiring experienced experts, lead to scalable, comparable results, and are 
time and cost-efficient than clinical interviews. Besides, the object is an individual's 
emotional experience, and therefore the subjectivity of self-reports makes sense to 
capture someone's psychological strain. However, self-report questionnaires face 
big shortcomings in the assessment of emotions over time. Most instruments (cov- 
ering the classification criteria of DSM-V and ICD-10) refer to a recall period of up 
to two weeks (see Table 10.2 for examples). Therefore, they are greatly biased by 
memory effects [96] and the motivation of an individual [97]. Studies are showing 
the questionnaire results are more negative when filled on Mondays while Saturday 
mornings are “the happiest moment during the week" and consequently provoke 
more positive test results [98, 99]. Other bias factors are fatigue and other non- 
assessed personal conditions (e.g., bad news occurring just that specific assessment 
day), as well as misunderstanding of the items and social desirability (aiming “to 
look good;" even in anonymous surveys) [100, 101]. This specific subjectivity is 
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wanted to a somewhat extent since it also informs the examiner about a person’s 
perception and interpretation. Nevertheless, the objectivity of assessment is always 
limited greatly by these interpretation biases. 


Physiology-Based Emotion Assessment 


As mentioned above, emotions are represented not only by the cognitive but also 
physical and expressive components. While the cognitive aspect of emotion cannot 
be observed directly, the physical and expressive aspects are often manifested in 
different bodily signals. An emotional state influences underlying human biology 
and psychophysiology, as well as the resulting behaviors. For example, stress can be 
manifested in hormonal changes; anxiety can be manifested in terms of a high pulse 
rate, while happiness can be expressed via laughter. As we point out in the following 
subsections, some of these manifestations are captured and modeled to determine 
the emotion in affective computing. 


Emotional Assessment from Blood 


The emotional state of the human may be assessed via blood-based analytics, as the 
emotional state influences the individual’s hormonal status. Especially in the field of 
biological psychiatry research, plasmatic biomarkers have been leading endeavors. 
The five most named plasmatic biomarkers (BDNF, TNF- alpha, IL-6, C-reactive 
protein, and cortisol) are classically used to predict psychiatric disorders like schizo- 
phrenia, major depressive disorder, or bipolar disorder [21]. In a meta-analysis, pat- 
terns of variation of those features were identified between those most important 
psychiatric disorders. The results indicated robust variations across studies but also 
showed similarities among disorders. The authors conclude that the implemented 
biomarkers may be interpreted as transdiagnostic systemic consequences of psychi- 
atric illness rather than diagnostic markers. This is in line with another review of 
Funalla et al. showing evidence for diagnostic biomarkers associated with obsessive- 
compulsive disorder (OCD) but did not show diagnostic specificity [102]. A com- 
monly used indicator for stress is the cortisol concentration found in human blood 
or saliva [103]. Overall, in this chapter, we do not focus on plasmatic biomarkers; 
we mention these for completeness. 


Emotional Assessment from Brain 


Other biological features to determine emotions with a rather high accuracy can be 
extracted from brain electroencephalography (EEG) [22]. With the help of EEG, 
the assessment of the brain’s summed electrical activity is enabled by recording 
the voltage fluctuations on the surface of the head. Emotion extraction is 
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consequently based on arousal and categorized into valence and excitation. EEG 
evaluation is traditionally performed by pattern recognition by the trained evalua- 
tor or by an automatic evaluation. For emotions, the patterns of alpha and beta 
waves are key classifiers. The alpha wave is associated with mild relaxation or 
relaxed alertness, with eyes closed (frequency range between 8 and 13 Hz). A beta 
wave has different causes and meanings and may occur during constant tensing of 
a muscle or during active concentration (frequency range between 13 and 30 Hz). 
According to Choppin (2000), high valence is associated with high beta power in 
the right parietal lobe and high alpha power in the brain’s frontal lobe [104]. High 
beta power in the parietal lobes is associated with higher arousal in emotions, 
while the alpha activity is lower but also located in the parietal lobes. More specifi- 
cally, negative emotions are represented by activity in the right frontal lobe, 
whereas positive emotions result in high power in the brain’s left frontal part. EEG 
was found to achieve 88.86% accuracy for four emotions: sad, scared, happy, and 
calm [105]. After assessing the EEG waves and extracting the particular emotional 
features, classifiers are trained for emotion identification. Popular is the Canonical 
Correlation Analysis (CCA) [106], Artificial Neural Network (ANN) [107], Fisher 
linear discriminant projection [24], and Adaptive Neuro-Fuzzy Interference 
System (ANFIS) [108]. Using K-Nearest Neighbor (KNN) [109], and Support 
Vector Machine (SVM) [105, 110] Mehmood and Lee (2016) used five frequency 
bands (beside alpha and beta, delta, theta, gamma waves) and identified the four 
emotions sad, scared, happy and calm with an accuracy rate of 55% (KNN) and 
58% (SVM). 

In a more clinical setting, Khodayari-Rostamabd [111] used EEG data to pre- 
dict the pharmaceutical treatment response of schizophrenic patients. A set of 
features was classified using the kernel partial least squares regression method to 
perform response prediction on the positive and negative syndrome scale (PANSS) 
with 85% accuracy. In another sample aiming to predict psychopharmacological 
treatment response (SSRI), the same research group extracted candidate features 
from the subject’s pre-treatment EEG using a mixture of factor analysis (MFA) 
model in a sample with patients suffering from depression [112]. The proposed 
method’s specificity is 80.9%, while sensitivity is 94.9%, for an overall predic- 
tion accuracy of 87.9%. Besides EEG, also functional Magnetic Resonance 
Imaging (fMRI) is used to assess emotions, especially by exploring the amygdala 
activity [113, 114]. Zhang et al. [114] analyzed connectivity change patterns in 
an fMRI data-driven approach in 334 healthy participants before and after induc- 
ing stress. Besides, the participants’ cortisol level was taken to classify pre- and 
post-stress states. The machine learning model revealed that the discrimination 
relied on activation in the dorsal anterior cingulate cortex, amygdala, posterior 
cingulate cortex, and precuneus and with a 75% accuracy rate. The advantage of 
using EEG for assessing emotions is that the data extraction is independent of 
facial or verbal expression that could be impaired due to, e.g., paraplegia, facial 
paralysis, but the necessity of a lab and the complex, costly installation and main- 
tenance of equipment is a big disadvantage not only for the practical field but also 
research projects [22]. 
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Emotional Assessment from Physiological Signal Collection 


A lot has been written about assessment of the emotional state of the individual 
from the physiological state—via, e.g., EEG (mentioned above), Electrocardiogram 
(ECG), Electromyography (EMG), Electrooculography (EOG), Galvanic Skin 
Response (GSR), Heart Rate (HR), Body Temperature (T), Blood Oxygen Saturation 
(OXY), Respiration Rate (RR), or Blood Pressure analytics (BP) [115-118]. From 
the technical perspective, the existing physiological signal-driven emotion assess- 
ment methods can also be divided into three categories (a) traditional machine 
learning methods, (b) deep neural network-based method, and (c) sequence- 
based models. 


Conventional Machine Learning Approach In the case of traditional machine 
learning-based approaches, first, a set of features are extracted from the captured 
data, and then different algorithms are used for model construction. Apart from the 
time domain characteristics, to leverage the spectral-domain characteristics such as 
power spectral density (PSD), spectral entropy (SE) is computed using Fast Fourier 
Transform (FFT) or Short-term Fourier Transform (STFT). 


SVM is probably the most widely used in physiological signal base emotion 
recognition among various machine learning algorithms. Das et al. extracted 
Welch’s PSD of ECG and Galvanic Skin Response (GSR) signals for emotion rec- 
ognition [119]. Liu et al. extracted a set of features from EEG and eye signals and 
used a linear SVM to determine three emotion states [120]. However, as regular 
SVM does not work in the imbalanced dataset, Liu et al. constructed an imbalanced 
support vector machine to solve the imbalanced dataset problem, which increased 
the punishment weight to the minority class and decreased the punishment weight 
majority class [121]. A few authors also used KNN (K = 4) to classify four emotions 
with the four features extracted from ECG, EMG, GSR, and RR [121]. In [115], the 
authors collected 14 features of 34 participants as they watch three sets of 10-min 
film clips eliciting fear, sadness, and neutrality, respectively. Analyses used sequen- 
tial backward selection and sequential forward selection to choose different feature 
sets for 5 classifiers (QDA, MLP, RBNF, KNN, and LDA). Wen et al. used RF to 
classify five emotional states with features extracted from OXY, GSR, and HR [122]. 


Deep Learning-Based Approach Among different deep  learning-based 
approaches, CNN is one of the most widely used. Martinez et al. trained an efficient 
deep convolution neural network (CNN) to classify four cognitive states (relaxation, 
anxiety, excitement, and fun) using skin conductance and blood volume pulse sig- 
nals [123]. Giao et al. used the Convolutional Neural Network (CNN) for feature 
abstraction from EEG signal [124]. In [125], several statistical features were 
extracted and sent to the CNN and DNN. Song et al. used dynamical graph convo- 
lutional neural networks (DGCNN), which could dynamically learn the intrinsic 
relationship between different EEG channels represented by an adjacency matrix to 
facilitate feature extraction [126]. DBN is also widely used for emotion recognition. 
It learns a deep input feature through pre-training. Zheng et al. introduced a recent 
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advanced deep belief network (DBN) with differential entropy features to classify 
two emotional categories (positive and negative) from EEG data, where a Hidden 
Markov Model (HMM) was integrated to accurately capture a more reliable emo- 
tional stage switching [127]. Huang et al. extracted a set of features and applied 
DBN in mapping the extracted feature to the higher-level characteristics space 
[128]. In the work of [129], instead of the manual feature extraction, the raw EEG, 
EMG, EOG, and GSR signals directly inputted to the DBN, where the high-level 
features according to the data distribution could be extracted. 


Sequence-Based Models To capture the temporal aspects of the physiological sig- 
nals, often sequence-based models are used. For example, Li et al. applied CNN 
first to extract features from EEG and then applied LSTM to train the classifier, 
where the classifier performance was relevant to the output of LSTM in each time 
step [130]. In the work of [131], an end-to-end structure was proposed, in which the 
raw EEG signals in 5 s-long segments were sent to the LSTM networks, in which 
autonomously learned features. Liu et al. proposed a model with two attention 
mechanisms based on multi-layer LSTM for the video and EEG signals, which 
combined temporal and band attentions [132]. 


Overall, as the captured signals are noisy, several pre-processing techniques are 
often used to eliminate the noise introduced from different sources such as cross- 
talk, measurement error, and instrument interference. The commonly used prepro- 
cessing techniques include filtering [133], Discrete Wavelet Transform (DWT) 
[134], Independent Component Analysis (ICA) [135], Empirical mode decomposi- 
tion (EMD) [136]. 

The overall psychophysiological approach for emotional assessment is cumber- 
some and not straightforward to be deployed in real-time for real-time accurate 
emotion recognition. Besides, the required complex laboratory setup (e.g., EEG) is 
time and cost-intensive. 


Expression-Based Emotion Assessment 


Following the cognitive and physical components of emotions, we focus on express- 
ing emotions in this section. We describe facial and verbal emotion recognition, as 
well as how posture may reflect human emotional states. 


Facial Emotion Recognition (FER) 
We broadly divide facial emotion recognition-related works into the following two 
groups (a) conventional FER approach and (b) deep learning-based FER approach. 


Conventional FER Approach For automatic FER systems, various types of con- 
ventional approaches have been studied. First, all these approaches assess the face 
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region and then extract a set of geometric features, appearance features, or a hybrid 
of geometric and appearance features on the target face. For the geometric features, 
the relationship between facial components is used to construct a feature vector for 
training [137, 138]. For example, Ghimire and Lee [138] used two types of geomet- 
ric features based on 52 facial landmark points’ position and angle. First, the angle 
and Euclidean distance between each pair of landmarks within a frame are calcu- 
lated. Second, the distance and angles are subtracted from the corresponding dis- 
tance and angles in the video sequence’s first frame. For the classifier, two approaches 
are used, using multi-class AdaBoost with dynamic time warping, and SVM on the 
boosted feature vectors. 


The appearance features are usually extracted from the global face region [139] 
or different face regions containing different types of information [140-143]. An 
example of using global features includes the exploration by Happy et al. [139]. The 
authors utilized a local binary pattern (LBP) histogram of different block sizes from 
a global face region as the feature vectors. They classified various facial expressions 
using a principal component analysis (PCA). This method's classification perfor- 
mance is poor as it cannot reflect local variations of the facial components to the 
feature vector. A few explorations also used features from different face regions as 
they may have different levels of importance, unlike a global-feature-based 
approach. For example, the eyes and mouth contain more information than the fore- 
head and cheek. Ghimire et al. [144] extracted region-specific appearance features 
by dividing the entire face region into domain-specific local regions. An incremen- 
tal search approach is used to identify important local regions, reducing feature 
vector size, and improving classification performance. 

For hybrid features, some approaches [144, 145] have combined geometric and 
appearance features to complement the two approaches' weaknesses and provide 
even better results in certain cases. 


Deep Learning-Based FER Approach The most adopted one in deep neural 
network-based FER is CNN. The main advantage is to completely remove or highly 
reduce the dependence on physics-based models and/or other pre-processing tech- 
niques by enabling "end-to-end" learning directly from input images [146]. Breuer 
and Kimmel [147] investigated the suitability of CNNs on different FER datasets 
and showed the capability of networks trained on emotion assessment and FER- 
related tasks. Jung et al. [148] used two different types of CNN: the first extracts 
temporal appearance features from the image sequences. The second CNN extracts 
the temporal geometry features from temporal facial landmark points. These two 
models are combined using a new integration method to boost the performance of 
facial expression recognition. 


However, as CNN-based methods are not suitable for capturing the temporal 
sequence, a hybrid approach combining both CNN (for spatial features) and 
LSTM (for temporal sequence) was developed. LSTM is a special type of RNN 
capable of learning long-term dependencies. Kahou et al. [149] proposed a hybrid 
RNNCNN framework for propagating information over a sequence using a 
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continuously valued hidden-layer representation. In this work, the authors pre- 
sented a complete system for the 2015 Emotion Recognition in the Wild (EmotiW) 
Challenge [150]. They proved that a hybrid CNN-RNN architecture for a facial 
expression analysis could outperform a previously applied CNN approach using 
temporal averaging for aggregation. Kim et al. [151] utilized representative 
expression-states (e.g., the onset, apex, and offset of expressions), specified in 
facial sequences regardless of the expression intensity. Hasani and Mahoor [152] 
proposed the 3D Inception-ResNet architecture followed by an LSTM unit that 
together extracts the spatial relations and temporal relations within the facial 
images between different frames in a video sequence. Graves et al. [153] used a 
recurrent network to consider the temporal dependencies present in the image 
sequences during classification. This study compared the performance of two 
types of LSTM (bidirectional LSTM and unidirectional LSTM) and proved that a 
bidirectional network provides significantly better performance than a unidirec- 
tional LSTM. 

In summary, hybrid CNN-LSTM (RNN) based FER approaches combine an 
LSTM with a deep hierarchical visual feature extractor such as a CNN model. 
Therefore, such a hybrid model can learn to recognize and synthesize temporal 
dynamics for tasks involving sequential images. Each visual feature determined 
through a CNN is passed to the corresponding LSTM, and it produces a fixed or 
variable-length vector representation. The outputs are then passed into a recurrent 
sequence-learning module. Finally, the predicted distribution is computed by apply- 
ing softmax [154, 155]. A limitation of this approach is the challenge of capturing 
the facial expression in real-time in daily life as a natural reaction to an emotional 
experience. Additionally, privacy issues can be a problem if a person does not want 
to be visually recorded during such intimate moments. 


Speech Based Emotion Recognition (SER) 


The existing literature on speech emotion recognition (SER) is also broadly divided 
into the following two categories (a) Conventional SER approach and (b) deep 
learning-based SER approach. 


Conventional SER Approach In traditional SER systems, there are mainly three 
steps—(a) signal pre-processing, (b) feature extraction, and (c) classification. At 
first, acoustic pre-processing such as denoising, segmentation is carried out to 
determine relevant units of the speech signal [156—158]. Once the pre-processing is 
done, several short-term characteristics of the signal such as energy, formants, and 
pitch are extracted, and short-term classification of the speech segment is done 
[159]. On the contrary, for long-term classification, mean, standard deviation is 
used [160]. Among prosodic features, the intensity, pitch, rate of spoken words, and 
variance play an important role in identifying various types of emotions from the 
input speech signal [161]. The relationship between different vocal parameters and 
their relation to emotion is often explored in SER. Parameters such as intensity, 
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pitch, and rate of spoken words, and quality of voice are frequently considered 
[162]. The intensity and pitch are often correlated to activation so that the value of 
intensity increases along with high pitch and vice versa [163, 164]. Factors that 
affect the mapping from acoustic variables to emotion include whether the speaker 
is acting, there are high speaker variations, and the individual’s mood or personality 
[165, 166]. 


In the existing SER literature, there are two types of classifiers—linear and non- 
linear. Linear classifiers usually perform classification based on object features with 
a linear arrangement of various objects [166]. In contrast, non-linear classifiers are 
utilized for object characterization in developing the non-linear weighted combina- 
tion of such objects [167—170]. The GMMs are utilized for the representation of the 
acoustic features of sound units. The HMMs, on the other hand, are utilized for 
dealing with temporal variations in speech signals [171]. 


Deep Learning-Based SER Approach In SER approaches, different types of 
deep neural networks are used. Senigallia et al. used a 2D CNN with Phoneme 
data as input data to determine 7 emotion states [172]. Zhao et al. combined Deep 
1D and 2D CNN for high-level learning features from input audio and log-mel 
spectrograms for emotion classification [173]. Convolutional Neural Network 
(CNN) also uses the layer-wise structure and can categorize the seven universal 
emotions from the defined speech spectrograms [174]. In [175], an SER technique 
based on spectrograms and deep CNN is presented. The model consists of three 
fully connected convolutional layers for extracting emotional features from the 
speech signal’s spectrogram images. Pablo et al. obtained emotional expressions 
that are spontaneous and can easily be classified into positive or negative [176]. 
Mao et al. trained the CNN to learn affect salient features and achieved robust 
emotion recognition with the variational speaker, language, and environ- 
ment [177]. 


The hybrid networks consisting of CNN and RNN are also used in SER [178— 
180]. This enables the model to obtain both frequency and temporal dependency in 
a given speech signal. Sometimes, a reconstruction-error-based RNN for continu- 
ous speech emotion recognition is also used [181]. SER algorithms based on CNNs 
and RNNs have been investigated in [180]. The deep hierarchical CNNs architec- 
ture for feature extraction has also been combined with LSTM network layers. It 
was found that CNN’s have a time-based distributed network that provides results 
with greater accuracy. Zhao et al. used a hybrid RCNN model to determine basic 
emotion [182]. Wootaek et al. used a deep hierarchical feature extraction architec- 
ture of CNNs combined with LSTM network layers for better emotion recognition 
[180]. Like FER, capturing the speech in real-time as a reaction to the emotional 
experience is a challenging issue. In addition to interfering with privacy, someone 
may express emotions without using expressed language because they are alone or 
do not want to speak, especially in emotional moments. Although individuals tend 
to adapt quickly to being observed, the awareness of being recorded might interfere 
with someone’s speech and expression of emotions. 
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Emotional Assessment from Posture 


In contrast to research on automatic emotion recognition focusing on facial expres- 
sions or physiological signals, little research has been done on exploiting body pos- 
tures. However, they can be useful for emotion recognition and even more accurate 
than facial features [23]. Bodily postures refer to the physical expression compo- 
nent of emotions and an important channel of communication. Since it is challeng- 
ing to categorize the expressed posture to discrete emotions due to the variety of 
validated emotion poses, researchers in this field focus first on defining features, 
aiming for understanding the cohesion and dimensional ratings with high validity 
[183]. Therefore, studies recorded actors displaying concrete, discrete emotions, 
and the recordings were then rated by the study participants categorizing these emo- 
tions. For example, Lopez et al. asked study participants to categorize emotion pos- 
tures depicting five emotions (joy, sadness, fear, anger, and disgust) and to rate 
valence and arousal for each emotion pose. Besides a successful categorization of 
all emotion categories, participants accurately identified multiple distinct poses 
within each emotion category. The dimensional rating of arousal and valence 
showed interesting overlaps and distinctions, increasing further granularity of dis- 
tinct emotions. Similarly, the Emotion Recognition Test (GERT) [85] was devel- 
oped to test the emotional recognition ability using video clips with sound 
simultaneously presenting facial, vocal, and emotional expression based on the pos- 
ture by using various data and video material and relatively large samples to vali- 
date. Individuals using the GERT test material are asked to watch video clips and 
rate the displayed emotions to assess their emotion recognition ability. 

Postures are also captured using movement sensor data from smartwatches or 
mobile phones [184]. Quiroz et al. observed the movement sensor logged by the 
smartwatches of 50 participants differentiated between the emotions happy, sad, 
and neutral as a response to an emotional stimulus in an experimental setting. 
Furthermore, the response was validated by additional data from the Positive Affect 
and Negative Affect Schedule Questionnaire (PANAS). Emotional states could be 
assessed well by self-report and data obtained from the smartwatch with high accu- 
racy across all users for classification of happy versus sad emotional states. Although 
the categories here depict only two emotions and other categorization difficulties 
need to be evaluated, movement sensors' usage still appears promising for emotion 
recognition purposes. Another smartphone-based approach also uses information 
about postures for recognizing emotions but uses self-reported body postures [185]. 
A mobile application was developed that classifies (based on the nearest neighbor 
algorithm) inserted poses into Ekman's six basic emotion categories and a neutral 
state. Emotion recognition accuracy was evaluated using poses reported by a sample 
of users. 

Although digital devices may be included to capture postures in real-time, this 
data collection is challenging to conduct in a person's daily life and close to their 
natural expression of emotions. Due to the necessity of leveraging, e.g., the use of 
cameras, this method is obtrusive and implies the same privacy issues we discussed 
in the context of FER and SER. 
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Internet-Use Based Emotion Assessment 


This subsection presents the emotion assessments based on the internet usage of an 
individual. Firstly, analysis on social networks (3.4.1) is discussed, followed by 
smartphone-based emotion assessment (3.4.2) and smartphone-based experience 
sample methods (3.4.3). 


Social Network Analysis 


Digital records of an individual’s behavior are extracted, including linguistic style, 
sentiment, online social networks, and other activity traces, which can be used to 
infer the individual’s psychological state. In particular, social networking platforms 
are becoming increasingly popular. They have recently been used more extensively 
to study emotions, as they are easily accessible to users, and researchers can collect 
the necessary information with the users’ consent. Based on this approach, Chen 
et al. aimed to identify users with depression or at risk of depression by assessing 
the individual’s expressed emotions from Twitter posts over time [36]. In another 
study, voluntarily shared Facebook Likes for N = 58,000 users were used to predict 
several highly sensitive personality attributes [37]: sexual orientation, ethnicity, 
religious and political views, personality traits, intelligence, happiness, use addic- 
tive substances, parental separation, age, and gender. All attributes were predicted 
with high accuracy, especially the ethnic origin and gender. Other emerging 
approaches focus on Spotify music or Instagram picture extraction as a feature to 
predict personality or mood outcomes [186]. 

Furthermore, studies focus on language content in social media networks or mes- 
saging systems to discover depressive symptoms, the so-called Natural Language 
Processing (NLP) [187]. The depressive language was characterized by more nega- 
tive and extreme words such as “always, everybody, never” [188]. The challenge 
with this data is that communication on Twitter, Facebook may be heavily distorted 
by aspects of social desirability and specific motivations that drive someone to 
express themselves on the Internet. Furthermore, data collection from someone’s 
account may face privacy issues. 


Smartphone-Based Emotion Assessment 


Smartphones are personal devices that individuals carry around with them almost 
all the time [189]. They include a plethora of onboard sensors (e.g., accelerometer, 
gyroscope, GPS) and can sense different user activities passively (e.g., mobility, app 
usage history) [190]. In this subsection, we review the smartphone-based methods 
for emotion assessment in its user’s natural daily environments. We consider the 
usage-based assessment methods, as well as touch-based ones. 
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Usage-Based Emotion Recognition Methods The smartphone provides numer- 
ous data sources for collecting real-world data about emotions. For example, defined 
FER and SER assessments can also be extracted from the smartphone's camera and 
microphone. Other smartphone-based sensing sources are connectivity (WIFI on/ 
off), smartphone status (screen, battery, power-saving mode), calls (type, duration), 
text messages (type, length), notifications (apps, category), calendar (initial query, 
logging of new entries), technical data (anonymized user ID, IP address, mobile 
phone type) [191]. Harari et al. [192] sensed conversations, phone calls, text mes- 
sages, and messaging and social media applications for individual trait assessment. 
Namely, they collected sensing data in five semantic categories (communication & 
social behavior, music listening behavior, app usage behavior, mobility, and general 
day- & night-time activity) and used a machine learning approach (random forest, 
elastic net) to predict personality traits.” MoodScope proposed to infer mood exploit- 
ing multiple information channels, such as SMS, email, phone call patterns, applica- 
tion usage, web browsing, and location [143]. In EmotionSense, Rachuri et al. used 
multiple Emotional Prosody Speech and Transcripts library features to train the 
emotion classifier [195]. In the same vein, researchers also demonstrated that aggre- 
gated features obtained from smartphone usage data could indicate the Big-Five 
personality traits [196]. We also find that there are multiple works, which use differ- 
ent information sources to infer the presence of a particular emotional state. For 
example, Pielot et al. tried to infer boredom from smartphone usage patterns like 
call details, sensor details, and others [197]. In their work on assessing stress, Lu 
et al. built a stress classification model using several acoustic features [198]. 
Similarly, Bogomolov et al. showed that daily happiness [199] and daily stress [200] 
could be inferred from mobile phone usage, personality traits, and weather data. 


Touch-Based Emotion Recognition Methods Widespread availability of touch- 
based devices and a steady increase [35] in the usage of instant messaging apps 
open a new possibility of inferring emotion from touch interactions. Therefore, 
research groups started to focus on typing patterns (Shapsough et al., 2016) using a 
built-in sensor (a smart keyboard) and using machine learning techniques to assess 
emotions based on different aspects of typing. For example, Lee et al. designed a 
Twitter client app and collected data from various onboard sensors, including typing 
(e.g., speed), to predict one user's emotion in the pilot study [201]. Similarly, Gao 
et al. used multiple finger-stroke-related features to identify different emotional 
states during touch-based gameplay [202]. Ciman et al. assessed stress conditions 
by analyzing multiple features of smartphone interaction, including swipe, scroll, 
and text input interactions [203]. Kim et al. [204] proposed an emotion recognition 
framework analyzing touch behavior during app usage, using 12 attributes from 3 


?Personality traits are generally described as relatively stable patterns of thought, feelings, and 
behaviors and are therefore well related to emotional theories [193, 194]. 
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onboard smartphone sensors. Although focused on narrow application scenarios, all 
of these works point to the value of touch patterns in emotion assessment. 


Smartphone-Based Experience Sampling Method Design 


One of the key requirements to develop a smartphone-based emotion assessment 
system is to collect emotion ground truth labels, which are typically collected as 
emotion self-reports by deploying an Experience Sampling Method (ESM), also 
known as an Ecological Momentary Assessment (EMA). The Experience Sampling 
Method (ESM) is a widely used tool in psychology and behavioral research for in- 
situ sampling of human behavior, thoughts, and feelings [5, 205]. The ubiquitous 
use of smartphones and wearable devices helps in the more flexible design of ESM, 
aptly termed as mobile ESM (mESM) [206-208]. It allows the collection of rich 
contextual information (e.g., sensor information, application usage data) along with 
behavioral data at an unprecedented scale and granularity. Frameworks like Device 
Analyzer, UbiqLog, AWARE, ACE, MobileMiner, or MQoL-Lab [190] have been 
designed to infer user's context based on sensor data, application usage details of 
the smartphones [142, 209—213]. While these frameworks help in the automatic log- 
ging of sensor data, self-reports related to various aspects of human life (like emo- 
tion) still require direct input from the user. 


Balancing Probing Rate and Self-Report Timeliness In the ESM studies, the 
participant burden mainly arises from repeatedly answering the same survey ques- 
tions. Time-based, event-based schedules are the most commonly used ESM sched- 
ules [214]. Time-based approaches aim to reduce probing rate (at the cost of fine 
granularity), while event-driven ones try to collect self-report timely (at the cost of 
a high probing rate). Recently, hybrid ESM schedules are designed combining 
time-based and event-based ones to trade-off between probing rate and self-report 
timeliness [215]. With the proliferation of smartphones, and other wearable devices, 
more intelligent and less intrusive survey schedules, including these limiting the 
maximum number of triggers, increasing the gap between two consecutive probes, 
have been designed. Several open-source software platforms, like ESP [216], my 
experience [217], psychology [218], Personal Analytics Companion [219], are 
available on different mobile computing platforms to cater to ESM experiments. 


Maintaining Response Quality Via Interruptibility-Aware Designs Recent 
advancements in interruptibility-aware notification management recommend sev- 
eral strategies to probe at opportune moments leveraging contextual information 
(e.g., placing between two activities like sitting and walking, after completing one 
task such as messaging or reading a text on mobile) [220—222]. In [223], the authors 
showed that features like the last survey response, phone's ringer mode, and user's 
proximity to the screen could predict whether the recipient will see a notification 
within a few minutes. Leveraging these findings, intelligent notification strategies 
were developed, which resulted in a higher compliance rate and improved response 
quality [224, 225]. However, one of the major challenges of using such details in 
mobile-based ESM design is resource overhead and privacy. Designing ESM sched- 
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Table 10.3 Summarization of the existing ESM design approaches 


Opportune 
ESM Schedule Probing rate Timeliness probing 
Time-based (TB) (e.g. [31], of x x 
[144]) 

Event-based (EB) (e.g. [194], 
[226]) 

Hybrid [214] 


Interruptibility-aware (e.g. 
[8], [220], [223], [227]) 
Ideal ESM schedule 


SI XN OX 
SIN S 
Sixx x 


ules based on the underlying study can overcome such limitations [226]. Ideally, an 
ESM schedule shall optimize the probing rate (like time-based schedules), reduce 
latency (like event-based schedules), and probing moments (like interruptibility- 
aware schedules) (Table 10.3). 


Pros and Cons in Different Emotion Assessment Approaches 


In the previous sections, we have presented different approaches to assess emo- 
tions. Table 10.4 summarizes the pros and cons of the different self-report and 
sensing methods of data collection. We pointed out that self-report questionnaires 
have the advantage of being rather a time and cost-efficient (for assessors) and 
enable to reveal cognitions that are otherwise hard to capture. Furthermore, the 
subjectivity of an individual’s view on their emotions expressed via self-reporting 
might be wanted in some contexts (e.g., for clinical diagnostics). However, self- 
reports are challenged by numerous confounding variables as fatigue, interpreta- 
tion and memory biases, non-assessed personal conditions, misunderstanding of 
the items, and social desirability [100, 101]. Some physiological assessment meth- 
ods might be more objective (like EEG or blood) but require a laboratory and 
complex setup and controlled environment. Due to this limitation, real-time 
assessment of emotions close to an individual’s everyday life experience is not 
possible. 

Additionally, some research is based on induced emotional states. Emotional 
reactions can be induced in experimental settings. However, the transfer and gener- 
alizability of such results into an individual’s real-life is doubtable. Besides, the 
period of data collection is often limited, and collecting a high volume of data from 
a large number of participants is difficult. Finally, the participants need to partici- 
pate actively and contribute to the data collection effort (via self-reports). 

In most cases, the data collection cannot be done passively and, consequently, 
lacks unobtrusiveness. Moreover, most of the discussed methods focus only on one 
data collection source (e.g., speech, EEG alpha waves, or social network analysis). 
They are, therefore, very limited regarding the complex emotional process described 
in the CPM [4]. 
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In the context of emotional assessment, novel, personal sensing methods embed- 
ded in daily life via wearables and smartphones promise to overcome some of those 
issues by providing real-time observations [143, 197, 207, 209-211]. These devices 
can capture data passively from different modalities without user intervention, log 
app usage behavior, and leverage different computational models on the device for 


Table 10.4 Pro’s and Con's of Emotion Assessment Methods 


Method Pro Con 
Questionnaire-based Emotion Assessment 


Self-report — Easy to conduct — Several biases (memory, interpretation, 
questionnaires — Access to cognitions social desirability) 
— Time-efficient for examiners | — Time consuming for participants (not 
— Established and accepted in | examiners) 
practical fields 


— Privacy is given 

— Standardization for the 
classification of individual 
results available 


l Physiology-based emotion assessment 


Blood — Objective assessment of — Intrusive 
autonomous nervous system — Not applicable outside a laboratory 
(ANS) reaction — Focus on a single factor 


— Standardization for the 
classification of individual 
results available 


Brain — Objective assessment of brain | — Not applicable outside a laboratory 
activity as emotional response | — High costs 
— Standardization for the — High setup up requirements 
classification of individual — Focus on a single factor 
results available 
Posture — Naturalistic expression — Categories vary greatly, no 
— Setup needed (cameras, standardization 
posture logging devices) — Interpretation of postures is limited 
— Important channel of — Focus on one class of emotion 
communication expression only 
Expression-based emotion assessment 
Facial emotion — Objective assessment of — Recording in real-time in daily life may 
recognition expression of emotions be difficult 
— No language needed — Privacy issues 
— Passive data collection — Camera needs to capture facial 
possible expression 


— Awareness of being recorded might 
interfere natural reaction 


Speech emotion | — No laboratory setup needed, | — Recording in real-time in daily life may 
recognition objective be difficult 

— No body language and — Privacy issues 

physical parameters needed — Awareness of being recorded might 

— Passive data collection interfere natural reaction 

possible — No standardization 


Smartphone—And internet-based emotion assessment 
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Table 10.4 (continued) 
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Method 


Pro 


Con 


Questionnaire-based Emotion Assessment 


Social network 


— Time and cost efficient 

— No special laboratory or 
technical equipment needed 
— Real-time behaviour 
accessible 

— Passive data collection 
possible 


— Only specific part of daily expressed 
emotions visible as embedded in a social 
network use context 

— Focusing on just a single class of 
behaviour 

— Biased by social desirability and other 
motivations to communicate on social 
platforms 

— Objectivity and/or standard values not 
given 


Ecological — No specific setup needed to — | — Self-report with potential biases (social 
momentary conduct desirability, interpretation bias) 
assessment — Unobtrusive — Data protection issues 

— Real time assessment in daily | — No standardization 

life 

— Access to cognitions 
Smartphone- — No specific setup needed to | — Data protection issues 


based sensing 


conduct 
— Unobtrusive 


— Perceived as “scary” by participants 
— No standardization 


— Real time assessment in daily 
life 

— Passive data collection 
possible 

— Great variety of data sources 
possible 

— Execute different models for 
emotion inference 


emotion inference. As a result, these devices are very promising to determine 
emotion-based behavior based on different usage patterns. However, the approach is 
still novel, and some participants are concerned about their privacy [227]. These 
concerns must be taken into account seriously but contrasted by the fact that per- 
sonal information is shared openly on the Internet nowadays. This ambiguity is 
labeled as Privacy Paradox and known since 2006 [228]. 


Tapsense: Smartphone Typing Based Emotion Assessment 


This chapter specifically focuses on assessing the individual's emotional state from 
the smartphone usage patterns via the authors' TapSense study. Therefore, in this 
section, we first describe the overall research approach of the keyboard interaction 
study. We focus on a typing-based emotion assessment scenario, which helps to 
identify the key requirements to design the emotion assessment model and the self- 
report collection approach using an Experience Sampling Method (ESM) (Sect. 
5.1.). In Sect. 5.2, the TapSense field study and data analysis are presented, and in 
Sect. 5.3. evaluated. In Sect. 5.4, the study is discussed. 
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Fig. 10.3 Overall 
approach of the TapSense 
requirements gathering, 
study design, data analysis, 
performance evaluation, 
and discussion 


Requirement 
Gathering 


Designing 
TapSense 


TapSense Study 


& Data Analysis 


TapSense 
Evaluation 


Discussion & 
Lessons Learnt 


Background 


The overall approach for the TapSense study is shown in Fig. 10.3. First, we gath- 
ered a set of requirements to design the keyboard interaction-based emotion assess- 
ment tool, followed by the actual design and implementation of the TapSense 
application. We discuss the study in detail and analyze the collected data. Finally, 
we evaluate the performance of the TapSense application and discuss the lessons 
learned from this study. 


Requirements 


TapSense study relies on the users’ smartphone usage patterns. We explain the 
scenario of typing-based emotion assessment in Fig. 10.4. As the user performs 
typing activity, we extract his/her typing sessions and the amount of time he/she 
stays in a single mobile application without. For example, when a user uses 
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WhatsApp without switching to other applications from t1 till t2, then we define 
elapsed time between t1 and t2 as a Typing Session. Once the user completes the 
session, he/she is probed via an ESM, i.e., emotion self-report, which is consid- 
ered the emotion ground truth. Later, several features are extracted from the typ- 
ing sessions and correlated with the emotion self-report to develop an emotion 
assessment model. This scenario suggests consideration of the following 
requirements, 


* Trace keyboard interaction for emotion assessment: The key requirements 
while determining emotions from the typing sessions is to make sure that (a) the 
typing details are captured correctly so that the relevant features can be extracted 
(b) the emotion ground truths are collected (ESM) and (b) an accurate emotion 
assessment model is constructed. We discuss these aspects further in this 
section. 

* ESM design for self-report collection: Probing a user after every session may 
induce fatigue due to many probes. So, the probing moments should be chosen 
in such a manner that it captures the user's response accurately (i.e., before it 
fades away from the user's memory) and at the same time, the probing rate is not 
too high. We discuss in detail the ESM design further in this section. 


€ Select your Emotion € Select your Emotion 


How are you feeling now? How are you feeling now? 
O Sad / Depressed O Sad / Depressed 

©) Happy / Excited ©) Happy / Excited 

© Stressed O) Stressed 

@ Relaxed @ Relaxed 

O No Response © No Response 

RECORD EMOTION RECORD EMOTION 

ESM Probe ESM Probe 


© / Q m/m 
HHH HH 


t1 t2 t3 t4 t5 t6 


Typing Session Typing Session 


Fig. 10.4 TaspSense Approach: Typing Session-based ESM-triggering scenario 
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Design and Implementation of TapSense 


TapSense consists of the following key components as in Fig. 10.5. TapLogger 
records the user’s typing activity. It implements a virtual keyboard for tracing key- 
board interactions. ProbeEngine runs on the phone to generate the user’s ESM noti- 
fications and collects the ESM responses. The typing details and the associated 
emotion self-reports are made available at the server via the Uploader module that 
synchronizes with the server occasionally, or, if the user is offline, once the user 
connects to the Internet. The emotion assessment model is constructed on the server- 
side to determine the different emotional states from the typing details and the emo- 
tion self-reports. In parallel, a set of typing features is also extracted to construct the 
inopportune moment assessment model, which feeds back the ProbeEngine to opti- 
mize the probe generation. Next, we discuss the two key components of TapSense 
(a) emotion assessment from keyboard interaction, (b) ESM design for the emotion 
self-report collection. 


TapLogger: Keyboard Interaction Collection 

The TapLogger module of TapSense implements an Input Method Editor (IME) 
[229] provided by Android OS, and we refer to it as the TapSense keyboard 
(Fig. 10.6). It is the same as any QWERT keyboard; it provides similar functional- 
ities as any Google keyboard. We have selected a standard keyboard because we 
aimed to provide similar functionalities. The user’s keyboard interaction experience 
does not deviate much from what he/she is used to. It differs from others, as it has 
the additional capability of logging user’s typing interactions, which, for security 
reasons, is not available in Google keyboard. To ensure user privacy, we do not store 
or record the characters typed. The logged information is the timestamp of each tap 
event, i.e., when a character is entered and the key input’s categorical type, such as 
an alphanumeric key or delete key. 


TapLogger 
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Fig. 10.5 TapSense High-level System Architecture 
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ProbeEngine: Emotion Self-Report Collection (ESM) 

The ProbeEngine module of TapSense issues the ESM self-report probes by deliv- 
ering a self-report questionnaire (Fig. 10.7). This survey questionnaire provides 
the option (happy, sad, stressed, relaxed) to record ground truth about the user’s 
emotion while typing. This captures four largely represented emotions from four 
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Fig. 10.8 Circumplex Model 


different quadrants of the Circumplex model [52], as shown in Fig. 10.8. We select 
these discrete emotions as their valence-arousal representation is unambiguous on 
the Circumplex plane. Any discrete emotion and its unambiguous representation 
on the valence-arousal plane are equivalents [230]. We also include the “No 
Response" option to select this option to indicate the current probing moment is 
inopportune. 


Emotion Assessment Model Construction 


The emotion assessment model in TapSense is responsible for determining the four 
emotion states based on the keyboard interaction pattern. This is implemented on 
the server-side once the typing interaction details and the emotion self-reports 
details are available. 
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Table 10.5 Features used to Construct TapSense Emotion Assessment Model 


Feature Name | Feature Description 

Session typing speed i Average of all ITDs present in the typing session E 
(MSI) 

Session length Number of characters typed in the typing session 

Session duration Time duration of the typing session 

Backspace percentage | Fraction of backspace and delete keys typed in the typing session 
Special character Fraction of special characters (non-alphanumeric) typed in the 
percentage typing session 

Last ESM trigger Emotion label as provided by the user 

response 


ITD elapsed time between two consecutive keypress events [ms] 


Emotion Assessment Features From raw data collected within every Typing 
Session, we extract a set of typing features as defined in Table 10.5. For every ses- 
sion, we compute the ITDs, i.e., the elapsed time between two consecutive keypress 
events for all the presses. We derive the mean of all ITDs in the session and use it as 
typing speed. We define it as the Mean Session ITD (MSI). We compute the back- 
space and delete keys present in a session and use it as a feature. This is used as the 
representation of typing mistakes made in a session. 


Similarly, we use the fraction of special characters in a session, session duration, 
and typed text length in a session as features. Any non-alphanumeric character is 
considered a special character. We use the last emotion self-report as a label for the 
model [215, 231]. However, at the later stage, when the TapSense model is opera- 
tional, we use the predicted emotion for the last session as the feature value for the 
current session. 


Emotion Assessment Model Trees-based machine learning approaches have been 
accurate in the context of emotion assessment in the past [201, 232]. We design a 
Random Forest (RF) based personalized multi-state emotion assessment model 
using the features described in Table 10.5 to assess the emotions. As typing patterns 
vary across individuals, derived, features will vary. Hence we construct a personalized 
model. We implement these models in Weka [233], building 100 Random Forest 
decision trees with a maximum depth of the tree set as ‘unlimited’ (i.e., the tree is 
constructed without pruning). We then derive the RF models' performance by deriv- 
ing the mean and variability of the accuracy for the 100 RF-based models. 


Experience Sampling Method Design 


The ESM used in TapSense is optimized in two phases. Phase 1 balances probing 
rate and timeliness of self-report collection, and Phase 2 tries to probe at the oppor- 
tune moments when the user's attention is available. We achieve this by designing a 
two-phase ESM [234]. We summarize it in Fig. 10.9. In Phase 1, we combine 
policy-based schedules to balance probing rate and timeliness and learn the 
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inopportune moment assessment model. In Phase 2, we make the inopportune 
moment assessment model operational. We discuss both phases in detail now. 


Phase 1: Balancing ESM Probing Frequency and Timeliness The collection of 
ESM emotion self-reports at the end of every typing session would help collect the 
labels close to the event, but it would lead to the generation of too many probes and 
user burden. To trade off these two conflicting requirements, we first assess the 
quality of the session itself, i.e., we make sure that there is a sufficient amount of 
typing done in a typing session for it to be considered. We issue the ESM probe only 
(a) if the user has performed a sufficient amount of typing, i.e., a minimum L = 80 
characters in a typing session, and (b) a minimum time interval, i.e., W = 30 min- 
utes has elapsed since the last ESM probe. To ensure the labels are collected close 
to the typing session, we use the polling interval parameter (T = 15 seconds) to 
check if the user has performed a sufficient amount of typing within a session. We 
describe the selection of threshold values based on initial field trials in Appendix 1. 
We name this ESM schedule the Low Interference High Fidelity (LIHF) ESM 
schedule (Fig. 10.9 (Phase 1)). 


Phase 2: Inopportune Moment Assessment Model As we collect self-reports, 
we obtain both “No Responses” and valid emotion responses. We leverage these 
labels to build the inopportune moment assessment model (Table 10.6). 


We use typing session duration and the typing length in a session as features 
since lengthy and longer typing sessions may indicate high user engagement and not 
be the ideal moment for triggering a probe. Besides, there may be some types of 
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Table 10.6 Features Used To Detect Inopportune Moments 
Feature Name = — | Feature Description 


Session duration - | Duration of the typing session —— mu 
Session length | Length of the text in the typing session I 
App category - | Category of the application 


Last ESM trigger response _| Last ESM trigger response 


applications like media, games when the users may not be interrupted for probing. 
So, we include the application type also as a feature. We categorize the applications 
into one of the 7 categories: Browsing, Email, Media, Instant Messaging (IM), 
Online Social Network (OSN), SMS, and ‘Misc,’ following the application's 
description in the Google Play Store. Moreover, we use the label of the last ESM 
probe response as a feature. We use it to determine whether the user continues to 
remain occupied in the current session and if he/she marked the previous session 
with “No Response.” However, once the model is operational and deployed, we use 
the predicted value of the inopportune moment for the last session as the current 
session’s feature value. Table 6summarizes the features used to implement the 
model. We construct a Random Forest-based prediction model to assess the inop- 
portune moments for all the users. The model is augmented with the LIHF schedule 
to assess and eliminate inopportune probes (Fig. 10.9 [Phase 2]). 


TapSense: Field Study and Data Analysis 


In this section, we discuss the TapSense field study and the dataset collected from 
the study. 


Study Participants 


We recruited 28 university students (22 males, 6 females, aged 24-35 years) to 
evaluate TapSense. We installed the application on their smartphones and instructed 
them to use it for 3 weeks. Three participants left the study in between, and the other 
three participants have recorded less than 40 labels. We have discarded these 6 users 
and collected data from the remaining 22 participants (18 males, 4 females). The 
ethics committee approved the study under the approval order IIT/SRIC/SAO/2017. 


Instruction and Study Procedure 
During the field study, we executed only Phase 1, where we implement the LIHF 


schedule for self-report collection. We instructed participants to select the 
TapSense keyboard as the default keyboard. We informed the participants that 
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when they switch from an application after completing typing activity, they may 
receive a survey questionnaire as a pop-up to record their emotions. We also 
advised the participants not to dismiss the pop-up if they are occupied; instead, 
they were asked to record “No Response” if they do not want to record emotion at 
that moment. 


Collected Dataset 


We have collected 4609 typing sessions during this study period, which consti- 
tute close to 200 hours of typing labeled with an emotional state of all the partici- 
pants (N = 22). Out of these sessions, we record 642 “No Response” sessions, 
which is nearly 14% of all recorded sessions. Notably, the actual number of ESM 
triggers is less than the number of typing sessions because, as per the LIHF 
policy, if two sessions are close (as defined by W in Fig. 10.8), only one ESM 
will be triggered to cater to both the sessions. We summarize the final dataset in 
Table 10.7. 


EMA Self-Report Analysis 


The users have reported two types of responses (a) One of the four valid emotions 
or (b) “No Response.” While the valid emotion labels are used to construct the emo- 
tion assessment model, the “No Response" labels are important to design the inop- 
portune moment assessment model for the ESM. 


Emotion Labels Analysis We show the distribution of different emotion states for 
every user in Fig. 10.10. We have observed that ‘relaxed’ is the most dominant emo- 
tional state for most of the users. Overall, we have acquired 14%, 9%, 30%, 47% 
sessions tagged with happy, sad, stressed, and relaxed emotion states. 


Table 10.7 Collected Data Summary 


Number of participants N =22 (18 m, 4 f) 
Total typing events 942,827 
Total typing sessions 4609 
Total typing duration (in Hr.) 199.1 
Mean typing sessions (per user) 209 (std. dev 167.2) 
Minimum number of typing sessions for a user | 46 
Maximum number of typing sessions for a 549 
user 
Total ESM triggers 2554 


Mean ESM trigger (per user) 116.1 (std. dev 71.9) 
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No Response Analysis We show the user-wise distribution of “No Response” 
sessions in Fig. 10.11a. Although for most users, the fraction of “No Response” 
labels is relatively low, for a few users, it is more than 4096. We observe the 
application-wise distribution of “No Response" sessions in Fig. 10.11b; the major- 
ity of the “No Response" labels are associated with Instant Messaging (IM) appli- 
cations like WhatsApp. We also compare the distribution of total “No Responsel 
and total valid emotion labels at weekday, weekend, working hour (9 am-9 pm), 
and non-working hour in Fig. 10.11c. We infer the working hour based on the 
timestamp of the ESM response. We compute the percentage of total "No 
Response;" and the percentage of total other sessions is recorded at these times. 
However, in our dataset, we do not observe any major differences among these 
distributions. We also explore the time-wise distribution of No Response sessions 
in Fig. 10.11d, which indicates that a small number of No Response sessions were 
recorded during the late-night from 3 am onwards. This can be attributed to overall 
less engagement during late night. 
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Fig. 10.10 Distribution of emotion labels for every user. All but 5 users (6,7,12, 13, 18) recorded 
all four emotions 
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Fig. 10.11 Distribution of No Response sessions (a) User-wise (b) App-wise (c) Time-wise (d) 
Hour-wise 


TapSense Evaluation 


In this section, first, we discuss the experiment setup. Then we evaluate the emotion 
classification performance and the ESM performance. Finally, we discuss the limi- 
tations of the study. 


Experimental Setup 


During the field study, we used the LIHF ESM schedule for collecting self-reports. 
However, to perform a comparative study across different policies, we require data 
from time-based and event-based ESM schedules under identical experimental con- 
ditions from every participant. In the actual deployment, identical conditions are 
impossible to repeat over different time frames. Hence, we generate traces for the 
other policy-based schedules from the data collected using LIHF ESM. We outline 
the generation steps for these traces in Appendix 2. We show the distribution of 
emotion labels obtained from different schedules after trace generation in Fig. 10.12. 


Baseline ESM Schedules 


Different ESM schedules, listed in Table 10.8, used for comparison are described. 


e Policy-based ESM: We focus on the Phase 1 approach (i.e., without optimizing 
the triggering) and use three policy-based ESM schedules— Time Based (TB), 
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Fig. 10.12 Frequency distribution of different emotions across various ESM schedules 


Table 10.8 Different ESM Schedules based on the Policy used in Phase 1 and Usage of the Model 


in Phase 2 
ESM Schedule Phase 1 Phase 2 
Time-based No model is used. 
EB Event-based No model is used. 
LIHF LIHF No model is used. 
TB-M Time-based Inopportune moment detection model 
EB-M Event-based Inopportune moment detection model 
LIHF-M LIHF Inopportune moment detection model 


Event-Based (EB), and LIHF. In the case of TB, probes are issued at a fixed 
interval (3 hours). In EB's case, after every typing session, a probe is issued 
while LIHF implements the LIHF policy. These approaches do not use an inop- 
portune moment assessment model. Comparing these schedules helps to under- 
stand their effectiveness in reducing the probing rate and collecting 
self-reports timely. 


* Model-based ESM: We use the following model-based ESM schedules—TB-M, 


EB-M, and LIHF-M. These ESM schedules implement TB, EB, and LIHF sched- 
ules in Phase 1, respectively, followed by the inopportune moment assessment 
model operational in Phase 2. In all these schedules, the model is constructed 
using the same set of features (Table 10.6) extracted from relevant trace (1.e., for 
TB-M, the model is constructed from the trace of TB and similarly). Comparison 
of these model-driven schedules helps to understand the efficacy of the model in 
assessing the inopportune moments and whether applying the model with any 
off-the-shelf ESM is good enough to improve survey response quality. 
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Overall Performance Metrics 


We use the classification accuracy to measure the emotion classification perfor- 
mance, and as for the ESM performance, we assess it along with the probing rate 
index and its reduction wrt. The classical self-report approach, timely self-report 
collection, inopportune moment identification, and valid response rate collection. 


Emotion Assessment: Classification Accuracy (Weighted AUCROC) The per- 
formance of supervised learning algorithms highly depends on the quality of labels 
[235]. The label quality can adversely impact classification accuracy [236, 237]. In 
our research, we use Typing Session emotion classification accuracy. We measure it 
in terms of the weighted average of AUCROC (auc,,,) using AUCROC from four 
different emotional states. Let f;, auc; indicate the fraction of samples and AUCROC 
for emotion state i respectively, then auc,,, = Ys € (happy, sad, stressed, relaxed] f; * AUC). 


ESM Performance Metrics 


Probe Frequency Index (PFI) We compare the probing frequencies of different 
ESM schedules using PFI, defined as follows. Let there be different ESM sched- 
ules (e € E) and N; denotes the number of probes issued for the user i for an 
ESM schedule e, then PFI for user i for ESM schedule e is 


N; 

Ve, max (N; ) l 

The Recency of Label (RoL) The timeliness of self-report response collection is 
measured using RoL defined as follows. Let there be different ESM schedules 
(e € E), and d; denotes average elapsed time between typing and probing for user 
i for an ESM schedule d then RoL for user i for ESM schedule e is 
expressed as, RoL; = 


expressed as, PFI? = 


Ve, max (d; ) 
Inopportune Moment Identification We measure Precision, Recall and F-score 
for inopportune moment assessment. We also compute the weighted AUCROC 
(auc,,,) for the inopportune and opportune moments. Let f;, auc; indicate the fraction 
of samples and | AUCROC for clas i respectively, then 
auc, = Ys € (inopportune, opportune Ti * auci. 


Valid Response Rate (VRR) We also compare the percentage of valid emotion 
labels for different ESM schedules. Let there be different ESM schedules (e € E), 
and nr, denotes the fraction of No Response sessions recorded for ESM e, then Valid 
Response Rate for ESM e is expressed as VRR, = (1 — nre) « 100. 
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Fig. 10.13 Emotion classification accuracy (AUCROC) using different ESM schedules 
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Fig. 10.14 Emotion classification (a) AUC,, for LIHF-M per user (b) Emotion-wise accu- 
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Emotion Assessment: Classification Performance 


The emotion classification accuracy for different ESMs is shown in Fig. 10.13. We 
observe that the LIHF-M outperforms other schedules with a mean AUCROC of 
78%. It returns a maximum improvement of 24% with respect to TB and an improve- 
ment of 596 with respect to EB. We also observe that after applying the inopportune 
moment assessment model, the mean AUCROC (auc,,,) improves (by 4%) for each 
corresponding schedule (TB, EB, LIHF). 

We also show the user-wise emotion assessment AUCROC (auc,,,) corresponding 
to the LIHF-M schedule in Fig. 10.14a. The quality of the prediction for each emotion 
category is presented in Fig. 10.14b. The emotion states are identified with an average 
f-score between 54% and 74%. We observe that the relaxed state is identified with the 
highest f-score, followed by sad, stressed, and happy states, respectively. As data vol- 
ume increases, as in the case of the relaxed state, the performance metrics improve. 
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Table 10.9 Ranking of Features Used to Construct TapSense Emotion Assessment Model 


Feature name Rank Average IG. 
Last ESM trigger response 1 0.468 
Session typing speed 2 0.376 

l Backspace percentage 3 0.270 
Session length 4 0.231 
Special character percentage |5 0.203 
Session duration 6 0.181 
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Fig. 10.15 Probing rate across ESM schedules: Average number of probes and the Probe 
Frequency Index. (a) Average number of probes (b) PFI comparison 


Influence of Emotion Assessment Features 

We find the importance of the input features used for emotion assessment using the 
*InfoGainAttributeEval' method from Weka. We compute the average Information 
Gain (IG) of every feature and rank them in Table 10.9. We observe that the last 
ESM response is the most discriminating feature, followed by features like typing 
speed and backspace percentage. All the features are found to have an input into the 
model for the emotion assessment. 


ESM Performance 


In this section, we evaluate the ESM’s performance in terms of the three parameters 
(ESM probing rate, self-report timeliness, and opportune probing moments). 


Probing Rate Reduction 

We compare the average number of probes issued by each ESM schedule in 
Fig. 10.15a. We observe that time-based ESM (TB) issues the minimum number of 
probes, event-based ESM (EB) issues the maximum number of probes, while LIHF 
ESM lies in between. It is observed that the average number of probes is reduced by 
64% for LIHF ESM policy. 

We also perform the user-wise comparison using the Probe Frequency Index 
(PFI) metric in Fig. 10.15b. For all users, PFI for LIHF ESM is lower than that of 
event-based ESM. Across all users, there is an average improvement of 54% in 
PFI. Time-based ESM is the best in PFI but does not capture self-reports timely, as 
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Fig. 10.16 Timeliness of the ESM collection across schedules: Average elapsed time and the 
Recency of Label. (a) Average elapse time (b) RoL comparison 


shown later. LIHF ESM schedule enforces a minimum elapsed time between two 
successive probes; it generates fewer probes and reduces probing rate compared to 
event-based ESM. 


Timely Self-Report Collection 

We measure how close to the event (i.e., typing session completion) the ESM sched- 
ule collects the self-report. We compare the average elapsed time between typing 
completion and self-report collection for different ESM schedules in Fig. 10.16a. 
The average elapsed time is the least for event-based ESM, highest for time-based 
ESM, while for the LIHF, it lies in between. The average elapsed time for label col- 
lection is reduced by 9% for LIHF. 

We also compare the recency of labels using RoL in Fig. 10.16b. We observe that 
for every user, RoL is minimum for EB, and for most of the users, RoL is maximum 
in the case of TB, while for LIHF, the RoL lies in between. In the case of EB, we 
issue the probe as soon as the typing event is completed; it can collect self-reports 
very close to the event, resulting in the lowest RoL. On the contrary, in TB, we per- 
form probing at an interval of 3 hours. As a result, there is often a large gap between 
typing completion and self-report collection, resulting in high RoL. However, in the 
case of LIHF, we keep accumulating events and separate two consecutive probes by 
at least half an hour; we compromise to some extent in the label recency, yet less 
than in the case of TB. 


Inopportune Moment Assessment 

We compare the inopportune moment classification performance of three model- 
based approaches in Fig. 10.17a. We observe that the LIHF-M attains an accuracy 
(AUCROC) of 89%, closely followed by EB-M (88%), while TB-M (75%) per- 
forms poorly. We also note the precision, recall, and F-score values of identifying 
inopportune moments in Fig. 10.17b using the LIHF-M schedule. We also report the 
recall rate of inopportune moments for every user in Fig. 10.17c. We observe that 
for 14% of the users, the recall rate is greater than 75%, and for 60% of the users, 
the recall rate is greater than 50%. It is observed that users with many “No Response” 
(Fig. 10.8a) get more benefit using the inopportune moment assessment model. In 
summary, the proposed model combined with LIHF ESM performs best, while 
other ESM schedules also assess the inopportune moments accurately with 
this model. 
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Fig. 10.17 Inopportune moment assessment performance (a) different ESM schedules (b) 
LIHF-M schedule (c) User-wise LIHF-M schedule 


Table 10.10 Ranking of Features Used to Construct Inopportune Moment Model 


Feature Name Rank Average IG 
Last ESM trigger response 1 10.669 

i App category 2 0.053 
Session length 3 0.019 
Session duration 4 0.012 


Influence of Inopportune Moment Assessment Features 

We find the importance of every feature by ranking them based on the information 
gain (IG) achieved by adding it for predicting the inopportune moment. We use the 
InfoGainAttributeEval method from Weka [233] to obtain the information gain of 
each feature. Our results (in Table 10.10) show that the last ESM probe response is 
the most important feature, followed by the application category. 


Valid Response Collection 

We compare the valid response rate (VRR) for LIHF, LIHF-M schedules in 
Fig. 10.18. We do not consider other schedules as those labels were generated syn- 
thetically. The VRR for LIHF is 86%, and the same for LIHF-M is 96%. This fur- 
ther proves the effectiveness of the inopportune moment assessment model. As the 
model is in place for LIHF-M, it assesses and skips probing at the inopportune 
moments, thereby improving the number of valid emotion responses.. 


TapSense Study Discussion and Lessons Learnt 


In our research, we leverage the user’s smartphone for accurate and timely emotion 
assessment. We designed, developed, and evaluated TapSense, which passively logs 
typing behavior and develops a personalized machine learning model for multi-state 
emotion detection. We log the keyboard interactions (typing patterns and not the 
actual content) of the user and infer four types of emotions (happy, sad, stressed, 
relaxed). We also proposed an intelligent ESM-based self-report collection method 
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and integrated the same with TapSense, optimizing the manual self-report collec- 
tion. We evaluate the emotion classification performance and the ESM performance 
of TapSense in a 3-week in-the-wild study involving 22 participants. The empirical 
analysis reveals that the TapSense can infer emotions with an average accuracy of 
78%. It also demonstrates the efficacy of the proposed ESM in terms of probing rate 
reduction (on avg. 24%), self-report timeliness (on avg. 9%), and probing at oppor- 
tune moments (on avg. 89%); all of which improve the emotion classification 
performance. 

However, a few factors need to be considered before deploying sensing technolo- 
gies as TapSense as an emotion assessment tool. First, it is crucial to consider key- 
board interaction experience should not be impacted while using the TapSense 
keyboard as most of the participants are conversant with the Google keyboard. 
However, we do not observe a significant effect in the app usage due to this, as we 
record 86% valid emotion labels and, on average, 209 typing sessions per user. 
Second, the model-driven probing strategy at opportune moments may not perform 
well for some users if the number of *No Response" labels are very few (less than 
4% of all sessions). If the number of no response labels is very less, then the model 
may not perform well and may not detect all the inopportune moments. 

Another factor to consider is which ESM strategy is to be adopted during self- 
report collection. We recommend using the LIHF strategy to reduce survey fatigue 
compared to fixed event-based (EB) schedules, but it may suffer from latency in the 
self-report collection. Time-driven schedules may not be used if long-time-interval 
separates two probes. This may miss capturing the fine-grain event details, which 
are likely to carry emotional signatures. Finally, during self-reporting, if the partici- 
pants have skipped the pop-up instead of selecting “No Response,” we could not 
capture those moments in our study. However, this can be easily incorporated by 
logging the skipping events. 

The study we have carried out involving TapSense has several limitations that 
may limit the results’ generalization. First of all, the study has been of small size, as 
only 22 users have been engaged in the study for only 3 weeks. On the other hand, 
given the number of self-reports and user typing sessions, we may assume we have 
captured a representative sample of the general population, with 25 minutes a day 
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of interaction being logged and labeled for an emotional expression. As we have 
shown, the representation of emotional states was diverse. The additional limitation 
may stem from the Android OS only study; the iOS participants may have different 
traits and emotional states. The emotional modeling may have led to different 
results. Nevertheless, we present the results as indicative and will follow future 
research in this context, given a larger population sample and longer study duration. 


Conclusive Remarks 


In this chapter, we focused on emotions as indicators of quality of life due to the 
association of positive/negative life experiences and positive/negative emotional 
states and other aspects of quality of life, such as health, safety, economic and men- 
tal well-being. We highlighted functions and the importance of emotions in a per- 
sons’ daily life and the challenge of assessing emotions in traditional vs. novel 
methods of assessing emotions. The advantages and disadvantages of the diverse 
methods are especially rooted in objectivity, required assessment setup, self-report 
bias, privacy, real-time measurement, obtrusiveness, and inclusion of a wide range 
of emotion components. 

In more depth, in this chapter, we have leveraged smartphone interactions to 
assess a user’s mental state. As smartphones have become a true companion of our 
daily life, they can passively sense the usage behavior and mental state. With numer- 
ous typing-based communication applications on a smartphone, typing characteris- 
tics provide a rich source to model user emotion. In the specific case, the users’ 
keyboard interactions predicted four different emotion categories with an average 
accuracy of 78%, confirming and outperforming other approaches of smartphone- 
based emotion-sensing technologies. Hence, emotional assessment is feasible to 
conduct via different technologies. 

TapSense study shows promising results for minimally obtrusive, smartphone- 
based emotional state assessment, which may be leveraged for further studies and, 
if largely improved, in clinical practice for a ‘companion assessment’ of individu- 
als’ mental health, accompanying the current gold standard assessment methods 
and approaches. It is in line with the recent research results showing a potential for 
co-calibration of the self-reported, gold-standard approaches with the technology- 
reported ones [238, 239]. The aspects of minimal obtrusiveness may be of interest, 
especially for the leaders in the self-assessment space—the co-called 
QuantifiedSelfers [190, 240] who leverage diverse self-assessment technologies 
for better self-knowledge and optimization of daily life activities for better well- 
being, health, and other outcomes in the long term. Overall, the TapSense may be 
seen as an example of the emerging Quality of Life Technologies [241, 242] to 
assess the individual’s behavioral patterns for better life quality. Once proven 
accurate and timely, and highly reliable in the context of daily life assessment, 
TapSense, and similar technologies may pave the way for technologies for a better 
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understanding of the physical, mental, and emotional well-being of populations 
at large. 


Appendices 
Appendix 1: Parameter Threshold Value 


We use three parameters L, W, T as defined in details in sect. 5.1 (Phase 1: Balancing 
ESM Probing Frequency and Timeliness) to balance between probing frequency 
and timeliness in label collection. L is defined as the minimum amount of typing 
performed in a typing session. W is defined as the minimum time elapsed since last 
ESM trigger, and T is defined as the polling interval (i.e. how frequently the typing 
session will be checked for sufficient amount of typing). Based on our initial data- 
set, we observe the CDF of session length (L) in Fig. 10.19a, which reveals that 
frequency distribution of session length is highly skewed. So, we select 66th percen- 
tile value as the threshold so that two-third values are less than this value. We 
observe similar CDF and frequency distribution (Fig. 10.19b) for inter-session gap 
(W) and use the 66th percentile value as the threshold. 

However, polling interval (T) is to be chosen in such a way that for most of the 
sessions, the event of interest is captured within this interval. In this case, the event 
is change of application after typing in a session. For this purpose, we measure the 
elapsed time between two successive key pressing events (ITD) in a session. We 
note the CDF of all ITD values from all sessions in Fig. 10.20. We observe that 9996 
of the inter-tap duration (ITDs) are less than 15 seconds i.e. for most of the sessions 
the application change happens after 15 seconds. So, we decide to use 15 seconds 
as the threshold for T. 


O 100 200 300, 400 | 


0 L 4 i i 
0 2K 4K 6K 8K 10K 0 100 200 300 400 500 
Session length (L) inter-session gap (W) in hour 


Fig. 10.19 CDFs for session length (L) and inter-session gap (W) reveal that the distribution is 
skewed. 66th percentile value is chosen as threshold so that two-third values are less than thresh- 
old. (a) CDF for session length (L). (b) CDF for inter-session gap (W) 
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Fig. 10.20 CDF of polling 
interval (T). It reveals the 
distribution is skewed and 
99% ITD values are close 
to 15 seconds 


0 5 10 15 20 25 30 
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Appendix 2: The ESM Trace Generation 


In this section, we discuss in detail the steps followed to generate trace for Time- 
based (TB) ESM and Event-based (EB) ESM schedule from the data collected using 
LIHF ESM schedule. In Fig. 10.21, a schematic is given to depict the same. Ei 
denotes the application switching event after sufficient typing. In case of LIHF 
ESM, there are 6 such events, however only 5 probes were issued (Fig. 10.21a). No 
probe is issued after E3 because it occurs within time-window (W = 30 minutes) 
since last probe (Probe 2). In order to generate the corresponding Time-based trace, 
probes are considered at 3 hour interval. As a result, there will be only one probe 
Probe 1 and all events E1 to E6 will be labeled with the single emotion response 
collected via it (Fig. 10.21b). But in case of conversion to Event-based ESM, all 
events are treated separately, as a result there will be in total 6 probes and the emo- 
tion labels will be assigned accordingly to the respective events (Fig. 10.21c). Next, 
we define the formal procedure for trace generation. 


Generation of Time-based Trace 

We take the trace collected from LIHF schedule »C.lihf p 5 as p _ 5 matrix where 
p denotes the total number of key press events. We generate the respective Time 
based trace »C.time p 5 following the Algorithm 1.We consider the sampling inter- 
val of Time-based ESM as 3 hours. We parse through (line 5-12) the LIHF trace 
»C. Lihf p. 5 and all key press events. As in case of LIHF, two responses may be 
recorded less than 3 hour interval, we may need to down-sample, which is per- 
formed in following way. If two emotion responses for key press events are col- 
lected within 3 hours, both are considered as a part of single session and the later is 
labeled with the previous emotion. Otherwise, they belong to different session and 
the new emotion response is considered (line 7—9). 


Generation of Event-based Trace 

We design Algorithm 2 to generate the corresponding event-based trace »C.event 
p. 5 from the collected LIHF trace »C. lihf p 5 .We consider changing application 
after typing as an event. We parse through (line 5-12) the trace obtained from LIHF 
schedule and all key press events. If two consecutive key press events are associated 
with different application, they belong to separate session (line 6-7). Otherwise, 
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a 


C Probe 1 C Probe 2 Probe 3 C Probe 4 a Probe 5 
E2 E5 E6 


E1 E3 E4 


t t+.5 t+1 t+1.5 t+2 t+2.5 t+3 t+3.5 
LIHF ESM probe schematic 


ET E2 ES E4 ES E6 
t t+.5 t+1 t+1.5 t+2 t+2.5 t+3 t-3.5 


Time-based ESM probe schematic 


Probe 3 
, - 
Probe 4 
C Probe 1 C Probe 2 ^ C Probe 5 C Probe 6 
E1 E2 ES E4 E5 E5 
t t+.5 t+1 t+1.5 t+2 t+2.5 t+3 t+3.5 


Event-based ESM probe schematic 


Fig. 10.21 Schematic to display how Event-based and Time-based traces are generated from 
LIHF ESM 


they are considered as part of the same session. In both these cases, no emotion 
response is dropped (unlike time-based), they are associated with different sessions. 
In case of LIHF, multiple sessions are grouped and tagged with single emotion, but 
in case of event-based schedule, this grouping is not done and every session is 
labeled with the same response. This is how the over-sampling is done in case of 
event-based schedule. 
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Algorithm 1: Time-based trace generation 


Input: [c] ^£, key pressing details of a user as obtained from LIHF schedule; Nite contain session 


px5 
lihf 
23 


number, (eye contain the application names, [C] .,, contain the timestamp of the key press, 


poy contain emotion recording timestamp for the session, roi’ contain the associated 
emotion,; p denotes the total number of key press performed by the user. 
Output: [C] fS": Corresponding time-based trace 
/* Copy the first row from LIHF schedule (session number, application name, emotion 
and other) */ 


session''"* — [C] ^£ (1, 1) 
apptime on [C]'/^/ (1, 2) 
emotion''"* — [C]'/^/ (1, 5) 


- =- 


4 (cy /"*(1,:) — [C]'/^/ (1, :) 
s fori —— 2to p do 
/* Find elapsed time between two consecutive emotion recording timestamp */ 
6 5 — time difference((C]'/^/ (i, 5),[C}!"F (i — 1, 5)) 
7 if 6 > 3 hours then . 
8 session''"* e session!" +1 
9 | emotion'"* — [C]'/^/ (i, 5) 
10 [C] ie (i, 1) — session'i"e 
m [C] /"* (i, 2 : 4) — [C] /^/ (i, 2 : 4) 
12 [C] /"* (i, 5) — emotion''"e 


13. return [C] iE 


Algorithm 2: Event-based trace generation 


Input: [cy i^f , key pressing details of a user as obtained from LIHF schedule; Ic contain session 


px5 
number, toys contain the application names, [c] ^^f contain the timestamp of the key press, 


[c] ^^f contain emotion recording timestamp for the session, [c] ^f contain the associated 


emotion,; p denotes the total number of key press performed by the user. 
Output: [C xS ; Corresponding event-based trace 
/* Copy the first row from LIHF schedule (session number, application name, emotion 
and other) */ 


session*"*"! e [C]'/^/ (1, 1) 
old app*"*"! e [C]'^£ (1,2) 
[C] "*"' (1, :) = [C] ^^^ (1, :) 
for i — 2 to p do 
/* Find current application name */ 
5 curr app*"*"! e [C]'/^/ (i, 2) 
6 if curr app*"*"' old app*"*"' then 
7 L session* "*"! — session* "*"! +1 
8 [C]* " *^' (i, 1) — session*"*"* 
» [C])*"*"' (j,2 : 5) — [C] "S (4,2 : 5) 


10 return [C]* Ver" 


awe d. = 
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Chapter 11 A) 
The Elusive Quantification of Self-Esteem: sx 
Current Challenges and Future Directions 


Stefano De Dominicis and Erica Molinario 


Introduction 


The concept of self-esteem has been central in the social-psychological literature 
since the late eighteenth century and it can be arguably considered one of the most 
important constructs in psychology. A quick database search of PsychINFO reveals 
a striking 52,126 results in March 2020, and 53,248 results in October 2020, show- 
ing how central this topic was and still is (with an estimate increase of 2000 hits per 
year) for scientists and practitioners alike. William James introduced this topic more 
than one hundred years ago, and more recently Rhodewalt and Tragakis [1, p. 66] 
stated that self-esteem is one of the “top three covariates in personality and social 
psychology research”. Perhaps, the relevance of self-esteem can be easily under- 
stood if we consider that this construct is linked to all levels of human existence, 
from mental illness to mental wellbeing: indeed, on the one hand, low self-esteem 
is related to various mental disorders, such as depression and anxiety (e.g., [2-5]); 
on the other hand, high self-esteem is related to various proxies of mental wellbe- 
ing, such as success, happiness, agency, and motivation (e.g., [6—8]). It is therefore 
fundamental to understand what self-esteem is, how it is assessed, and why it is so 
important for people’s quality of life. 
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Definition of Self-Esteem 


In a way, we all know what self-esteem “really is”: indeed, we can have a fairly 
good understanding of what is meant by self-esteem through introspection and 
observation of the behavior of others [9]. After all, self-esteem is a human phenom- 
enon; yet, it is hard to put that understanding into precise words. In fact, as soon as 
we begin to examine self-esteem more closely, the understanding of this construct 
becomes quite problematic. The issue of defining self-esteem is crucial because 
definitions help shape what to focus on, which methods to choose and use, and what 
standards should be adopted to accept or reject evidence or conclusions [10]. 
Nevertheless, to find a concise and overarching definition of self-esteem is challeng- 
ing because it encompasses different aspects and levels of analysis related to the 
context and its time stability or fluctuation. Self-esteem can be understood in terms 
of values (such as self-enhancement and openness to change values; [11]), feelings 
or affective dimensions (such as pride and shame; [12]), motivational (such as the 
desire to protect, maintain, and enhance feelings of self-worth; [13]), cognitive 
(such as evaluative components of self; [14]) and behavioral factors (such as being 
more independent or assertive, or more willing to exercise to gain fitness instead of 
reducing dissatisfaction with one’s body image; [6, 15, 16]). 

Historically, self-esteem has been conceptualized in terms of (a) self-competence, 
or the ratio of a person’s successes over her failures in areas of life that are relevant 
to personal identity (dating back to William James work 1890/1983; James, 
Burkhardt, Bowers, Skrupskelis, and James, 1981; [17]) and (b) self-worth, or the 
affect concerning the degree to which one feels good about oneself [18]. Therefore, 
self-esteem is considered an evaluative psychological process that reflects both the 
extent to which people accept and like themselves, and believe they are competent 
(e.g., [19, 20]). Such evaluation can occur in relation of a specific point on time 
(state self-esteem) or as an overall and more stable evaluation of the self (global 
self-esteem). Scholars have suggested that this evaluation may involve the assess- 
ment of several dimensions of the self. For instance, Tafarodi and Swann [20, 21] 
suggested that self-esteem involves perceptions of self-worth (i.e., self-liking) and 
personal efficacy and self-regard of one’s capabilities (i.e., self-competence); or as 
suggested by O’Brien and Epstein [22], it encompasses several dimensions related 
to worthiness, competence, and global self-esteem. 

Favorable views of the self, vs. evaluations of the self that are either uncertain 
or negative would then be the manifestations of high vs. low self-esteem, respec- 
tively [23]. It is relevant to note that self-esteem reflects subjective perceptions 
rather than objective reality, and therefore could be either accurate or not [24]. We 
suggest that, to fully understand self-esteem, it should be conceptualized in its 
social-psychological context. Therefore, we define self-esteem as the extent to 
which one person accepts and likes herself (in a specific point on time or overall) 
according to socially and personally defined standards, as well as believes of being 
competent in specific areas of life which are relevant to her personal and social 
identity. 
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To further corroborate this perspective, we should highlight that our definition is 
in line with sociometer theory [25], which basically consider self-esteem as the 
output of a psychological meter, or instrument, that monitors the quality of people’s 
relationships with others [26]. This psychological instrument is used by the self to 
monitor and respond to threats to the basic need to belong—i.e., that innate human 
need to form and maintain at least a minimum quantity of interpersonal relation- 
ships [25]. What follows, is that the degree to which a person perceives others to 
regard their relationship with herself as valuable, important, or close—i.e., her own 
relational evaluation—might change in specific situations or through the lifespan. 
Accordingly, when a person's relational evaluation is changing, the sociometer puts 
her attention to the related social acceptance threat or reward and motivates her to 
deal with it. Thus, the sociometer output is the affectively-charged self-appraisal 
that we typically perceive as self-esteem [27]: 


At its core, self-esteem is one's subjective appraisal of how one is faring with regard to 
being a valuable, viable, and sought-after member of the groups and relationships to which 
one belongs and aspires to belong. [26, p. 2] 


This understanding of self-esteem appears coherent and comprehensive, as it can 
indeed explain why self-esteem is a relatively stable, but by no means immutable, 
psychological trait, as well as why it appears that self-esteem trait might have a 
specific trajectory across the individual's lifespan [28]. 

In light of the above-mentioned arguments, the assessment of self-esteem 
becomes a critical issue because it lies at the heart of empirical research. Accordingly, 
the focus of this chapter is to synthetize previous work detailing the assessment of 
self-esteem and to link this work with future possibilities, especially those arising 
from the beginning of the digital age. 


Assessing Self-Esteem 


With more than 200 different scales that ostensibly assess self-esteem [29], the criti- 
cal issue of assessing self-esteem is far from simple. One of the reasons of this 
enormous effort lies in a critical element of self-esteem, namely the fact that self- 
esteem is, by definition, a subjective construct which is not tied to objective stan- 
dards [30]. Indeed, because its subjective nature, self-esteem has been assessed 
mostly by self-report scales [31], and to a smaller extent by implicit measures [32]. 
It is based on subjective, affective-laden evaluations of one's own self, which how- 
ever can occur with respect to specific domains (such as work, athletics or physical 
appearance) or to more broad and general level (such as overarching evaluation of 
the self as a whole). 

A time specific self-evaluations represent the state self-esteem, which refers to 
the "feeling" aspect of self-esteem, that is an individual's affectively loaded self- 
evaluation in a given, specific situation [26]. Whereas, evaluation of the self as a 
whole refers to global, general or trait self-esteem [31]: it is one's long-term, 
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characteristic and somewhat stable affectively charged self-evaluation. Because of 
trait self-esteem is a person’s “summary” self-evaluation, it may or may not reflect 
her state self-esteem in a particular situation. 

Finally, self-esteem may capture self-evaluations in specific domains: fluctua- 
tions of state self-esteem around a person’s self-esteem typical level can be under- 
stood in terms of contingencies of self-worth [33]. Central to this model is the 
controversy that the way events and circumstances impact self-esteem lies on the 
perceived relevance of those events and circumstances to one’s contingencies of 
self-worth: in other words, contingency self-worth represents a self-evaluation par- 
ticularly vulnerable in which a failure or rejection is devastating to our sense of 
self-esteem [13, 34]. 

Historically, scholars involved in self-esteem research have focused mainly on 
individual differences in dispositional self-esteem [26]. However, from a practitio- 
ner’s perspective, it is worth studying both stable self-esteem and its fluctuations, 
not only for the obvious advantage of better defining the construct itself, but also 
due to its possible applications. From the clinical perspective, measuring fluctuation 
of self-esteem allows to assess the efficacy of clinical interventions quantifying 
changes in self-esteem. Additionally, it can be used as a valid manipulation check in 
experimental design in which self-esteem is enhanced or diminished in a specific 
point in time. Finally, it can be used to understand confounded relations with other 
constructs. In the subsections below we discuss the assessment of both trait and 
state self-esteem (2.1), as well as situational self-esteem (2.2). In Table 11.1 we 
report the main characteristics of the reviewed scales measuring self-esteem. 


Trait, Global, and State Self-Esteem Measures 


As James [37] argues self-esteem is open to momentary changes, thus it raises and 
falls as a function of one’s aspirations and success experiences. Within this frame- 
work—developed from the wide body of research that supports self-esteem as an 
enduring yet flexible concept [18, 38-40]—-state self-esteem refers to how we eval- 
uate or feel about ourselves in a given situation or at a given point in time. State 
self-esteem is therefore considered as a series of transitory states, momentary fluc- 
tuations, short-lived changes in one's own global self-esteem [36]. However, 
although momentary self-evaluations may be context dependent, there is a self- 
feelings people have the tendency to maintain and a level of self-esteem that derives 
by averaging feelings about themselves at one time across a number of different 
social situations: trait or global self-esteem. 


Trait and Global Self-Esteem Measures 


To assess trait self-esteem, scholars have developed several measures which differ 
in number of items and latent dimensionality. Here we discuss the Single-Item Self- 
Esteem Scale (SISE; [35]), the Rosenberg Self-Esteem Scale (RSE; [18]), and the 
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Self-Liking/Self-Competence Scale-Revised (SLSC-R; [20, 21]) which are among 
the most common tools used to assess trait self-esteem. 

Single-Item Self-Esteem Scale (SISE). The simplest way to assess trait self- 
esteem is to ask individuals whether they have high self-esteem. In fact, this is the 
core of the Single Single-Item Self-Esteem Scale, which aims to assess the global 
self-worth or the overall attitude that one holds about oneself (SISE; [35]). The 
SISE was developed from the Rosenberg Self-Esteem Scale (described in the next 
section of this chapter) to assess self-esteem in contexts where time or other con- 
straints severely limit the possibility of using or administering more complex or 
comprehensive measures of self-esteem. The SISE, as suggested by its name, con- 
sists of a single item and assesses a person’s explicit knowledge about her global 
self-evaluation. This very brief, standardized measure of global self-esteem is valid 


Table 11.1 Measures of self-esteem reviewed in this chapter 


Number of 
items and Type of 
Response Definition and Components of self- 
scale self-esteem esteem 


] item Global self-worth: Overall Trait/ 
5-point attitude that one holds about Global 
Likert scale | oneself 
10 items Global self-worth: Overall Trait/ 
4-point attitude that one holds about Global 
Guttman oneself 
scale or 
5-point 
Likert scale 


Name of the 
measure References 


Single-Item 
Self-Esteem Scale 
(SISE) 

Rosenberg 
Self-Esteem Scale 
(RSE) 


Self-Liking/ [21] 
Self-Competence 
Scale-Revised 
(SLSC-R) 


16 items Self-esteem involves a personal Trait/ 
5-point sense of worth (self-liking) and a | Global 
Likert scale | personal sense of efficacy 
(self-competence): 

(a) Self-liking: feeling positive 
towards one’s own self; 

(b) Self-competence: feeling 
capable and in control, and to 
believe that one will be successful 
in the future 


State Self-Esteem [36] 
Scale (SSES) 


20 items Self-esteem is the overall State 
5-point evaluation of the self and 

Likert scale | comprises: 

(a) Performance self-esteem: the 
extent to which a person feels her 
performance is worthy; 

(b) Social self-esteem: the extent 
to which a person feels self- 
conscious, foolish, or embarrassed 
about her public image; 

(c) Appearance self-esteem: the 
extent to which a person feels 
about her physical appearance. 
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Name of the 
measure 


Multidimensional 
Self-Esteem 
Inventory (MSEI) 


Number of 

items and 

Response 
References | scale 


[22] 116 items 
5-point 
Likert scale 


Definition and Components of 
self-esteem 


Global self-esteem (satisfaction 
with the self and confidence); 
identity integration (the self’s 
internal integrity); and Defensive 
self-esteem (defensive reaction of 
changes in one’s own 
self-esteem). 

Components of self-esteem: 

(a) Lovability (ability to express 
and receive affection) 

(b) Likeability (feeling accepted 
and liked by others) 

(c) Moral self-approval 
(satisfaction with one’s moral 
values and acting accordingly) 
(d) Body appearance/physical 
attractiveness (being satisfied with 
one’s body image) 

(e) Competence (ability to master 
new tasks) 

(f) Personal power (being 
assertive and able to influence 
others) 

(g) Self-control (being 
disciplined, persistent, able to set, 
and reach one’s goals) 

(h) Body functioning/vitality 
(motor coordination and the 
feeling of being fit) 


Type of 
self- 
esteem 
Trait/ 
Global and 
Specific 


i Contingency of 
Self-Worth scale 
(CSWs) 


[33] 35 items 
7-point 


Likert scale 


Seven domains of contingent 
self-worth: 

1. Academic competence 

2. Physical appearance 

3. Virtue 

4. Having God’s love 

5. Having love and support from 
family 

6. Outdoing others in competition 
7. Obtaining others’ approval 


Contingent 


and reliable [31] and asks participants to rate whether to have high self-esteem (‘J 
have high self-esteem") is true for them (on a 5-point Likert scale, ranging from ‘not 
very true of me’ to ‘very true of me.’). Although very simple to use, single-items 
measures are often less reliable than other multi-items measures, especially when 
constructs are heterogeneous [41]. Specifically within the realm of self-esteem and 
compared to multi-item measures, single-item measures are more susceptible to a 
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person’s biased knowledge of her own explicit feelings of specific or global self- 
worth, and are also more vulnerable to acquiescence and social desirability [31, 35]. 
Yet, SISE demonstrated to have a high convergent validity with other measures of 
self-esteem [31, 36] and therefore can be used without reservations in situations and 
research contexts that would require a single-item measure of self-esteem. 

Rosenberg Self-Esteem Scale (RSE). Perhaps the most commonly used scale to 
assess trait self-esteem is the Rosenberg Self-Esteem Scale (RSE; [18]). The global 
self-esteem measured by the RSE is defined as the overall attitude one holds about 
oneself. This mono-dimensional definition of self-esteem implies the assumption 
that one might believe to be ‘good enough’ (high self-esteem) or not—meaning, to 
occur in self-rejection and to lack self-respect. This 10-item, easy-to-administer, 
self-esteem scale (example question: “J feel that I have a number of good quali- 
ties"), is originally designed as a 4-point Guttman scale but is often measured on a 
5-point Likert-type scale, ranging from (1) Strongly disagree to (5) Strongly agree. 
It quickly became the “gold standard” for self-esteem research [20]. Although RSE 
is the most widely used assessment of self-esteem in research with a high reliability 
and validity [31], there has been discussions about its mono-factorial or multi- 
factorial structure [42]. However, it seems that a prominent global self-esteem fac- 
tor consistently explains a considerable amount of variance in the RSE items 
[43-45], supporting the hypothesis of the mono-dimensionality of the RSE 
items [46]. 

Self-Liking/Self-Competence Scale-Revised (SLSC-R). To solve the single- vs. 
multiple-factor composition of the measures of self-esteem, a specific scale measur- 
ing two distinct components of global self-esteem was developed. Specifically, the 
Self-Liking/Self-Competence Scale (SLSC; and its Revised version SLSC-R) mea- 
sures the two dimensions of self-esteem corresponding to a personal sense of worth 
(i.e., self-liking) and to a sense of personal efficacy (i.e., self-competence; [20, 21]). 
More specifically, the authors define self-liking as feeling positive towards one’s 
own self, while self-competence refers to feeling capable and in control, and to 
believe that one will be successful in the future. The SLSC is a 16-item self-report 
scale, measured on a 5-point Likert-type response scale, which encompasses two 
8-item subscales measuring each of the two components of self-esteem. The two 
dimensions (self-liking and self-competence) are related, but substantially distinct 
[31]. This scale has shown high reliability and convergent, discriminant and con- 
struct validity. 


State Self-Esteem Measures 


As mentioned, a momentary self-evaluation is represented by the state self-esteem, 
also called self-esteem feeling, which indicates a person’s affectively laden self- 
assessment in a given situation [26]. The development of a measure of state self- 
esteem stemmed by the need for an instrument designed exclusively for assessing 
the momentary self-evaluations and self-esteem fluctuation [36, 47]. Here we 
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discuss the State Self-Esteem Scale (SSES, [36]) and the Multidimensional Self- 
Esteem Inventory (MSEI; [22]). 

State Self-Esteem Scale (SSES). The State Self-Esteem Scale (SSES) is a 20-item 
Likert-type scale designed for measuring temporary changes in individual self- 
esteem and is composed by three subscales including performance, social, and 
appearance self-esteem [36]: the performance component measures the extent to 
which subjects feel their performance is worthy (example question: “I feel confident 
about my abilities"); the social factor assesses the extent to which people feel self- 
conscious, foolish, or embarrassed about their public image (example question: “I 
feel self-conscious" [reversed item]); finally, the appearance element is instead 
related to physical appearance (example question: “I feel good about myself"). 

The Multidimensional Self-Esteem Inventory (MSEI; [22]) is an extensive self- 
report inventory which aims to define a respondent's profile across eight categories: 
four categories related to worthiness (lovability, likability, moral self-approval, and 
body appearance) and four related to competence (competence, personal power, 
self-control, and body functioning)—which are understood to impact self-esteem. 
Additionally, the MSEI includes three dimensions related to global measures of 
self-esteem, sense of identity, and defensiveness—which are understood as overall 
characteristics of or related to self-esteem. This inventory, initially developed as a 
clinical test for measuring and treating self-esteem and self-esteem related issues 
[24], has been widely used and validated across a great variety of domains both in 
research and clinical work (e.g., abuse, substance abuse and harassment; positive 
psychology, adjustment and emotional intelligence; academic, work and sport per- 
formance; goal attainment; childhood and adolescence wellbeing; emotional aware- 
ness, expression, reactivity and regulation; health psychology and physical 
wellbeing; mood, eating and personality disorders; stress and coping, trauma, anxi- 
ety; treatment, prevention, psychotherapy, self-help; for a complete list see [48]). 

The MSEI consists of 116-item rated on a 5-point Likert scale: (‘strongly agree’ 
to ‘strongly disagree’; or, “hardly ever’ to ‘very often’) which profiles the respon- 
dent’s self-esteem on the following 11 scales: lovability (ability to express and 
receive affection), likeability (feeling accepted and liked by others) moral self- 
approval (satisfaction with one’s moral values and acting accordingly), body appear- 
ance/physical attractiveness (being satisfied with one’s body image), competence 
(ability to master new tasks), personal power (being assertive and able to influence 
others), self-control (being disciplined, persistent, able to set, and reach one’s 
goals), body functioning/vitality (motor coordination and the feeling of being fit), 
global self-esteem (satisfaction with the self and confidence), identity integration 
(the self’s internal integrity), and defensive self-esteem (defensive reaction of 
increasing one’s self-esteem). This instrument was developed by gathering and cat- 
egorizing thousands of incidents that participants reported as impacting on their 
self-esteem [24]. Accordingly, the emerged eight areas of life impacting self-esteem 
can be considered the components of self-esteem. In addition, the global measures 
of self-esteem and of secure sense of identity provide information about the overall 
feelings of worthiness and sense of security in one’s identity; and the assessment of 
defensive self-enhancement differentiates between secure vs insecure self-esteem: 
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high scores express a biased self-presentation which denies weaknesses and claims 
strengths and which may not correspond to genuine self-evaluations [24]. 

The MSEI can be compared to other measures of self-esteem (especially with the 
SLSC-R) because the eight components can be construed as corresponding to the 
realms of worthiness or competence. Specifically, the categories of lovability, lik- 
ability, moral self-approval, and body appearance imply that the source of self- 
esteem is more dependent on acceptance and worth, and in fact they are based in 
part on a judgment of acceptance and value (worthiness). Instead, the categories of 
competence, personal power, self-control, and body functioning imply that the 
source of self-esteem is more dependent on one’s own ability to proactively impact 
the world, and are in fact based in part of one’s own agency and efficacy (compe- 
tence)—namely, the pillars of perceived self-efficacy [49]. However, despite the 
validity, applicability and the comprehensiveness of the MSEI [10, 24, 48, 50], it is 
not always possible to use such instrument out of clinical settings where time con- 
strains or other impediments might hinder the likelihood of its usage [22]. 

Nevertheless, it is worth noting that MSEI is able to provide important insights 
that other instruments cannot grasp. First, global measures of self-esteem may not 
provide sufficient information specificity on which facet or dimension of self- 
esteem is particularly problematic (or functional) for a given person. Measures that 
are too specific or contingent are also problematic since a particular domain (e.g., 
athletic performance) may be relevant for someone but not for others [26]. Therefore, 
the mid-level components of self-esteem identified by the MSEI represent a level of 
analysis useful for effective applied interventions: on the one hand, these compo- 
nents are general enough to be related to a general trait measure of self-esteem (to 
which they have strong correlations; [24]) and thus changes in these component can 
have an impact on overall self-esteem; on the other hand, they are specific enough 
to provide a deeper, idiosyncratic understanding about how to build on one's own 
strengths and overcome weaknesses in the likelihood of an intervention for increas- 
ing self-esteem [24]. 

Second, the dimension of defensiveness assessed by the MSEI is a critical ele- 
ment in assessing self-esteem: it represents a person's social desirability bias, 
namely a bias in self-appraisal, in which she claims rare virtues and denies common 
human weaknesses (e.g., gladly accepting criticism vs. never trying to avoid 
unpleasant responsibilities). Evaluating defensiveness scores on the MSEI allows 
insight into self-presentation and projection to others of an overly positive image of 
themselves, or in other words of a false self-esteem. 


Contingency of Self-Esteem 


Expanding upon James’ [37] idea that individuals differ in the domains on which 
they base their self-esteem, scholars have proposed several measures that capture 
domain-specific self-worth. The domains represent the context in which we are not 
only most likely to pursue self-esteem, but also are most vulnerable—in which a 
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failure or rejection is devastating to our sense of self-worth [34]. Based on the work 
carried out by Crocker, Luhtanen, Cooper, and Bouvrette [51], which we discuss 
next, researchers have developed measures of contingent self-worth in several 
domains, such as friendships [52], romantic relationships [53, 54], and body 
weight [55]. 

Contingency of Self-Worth scale (CSWs). According to Crocker and Wolfe [33], 
individuals differ in the areas on which they base their self-worth and they proposed 
a model that examined self-worth in specific domains. CSWs is a 35-item measure 
divided into seven domains of contingent self-worth: academic competence, physi- 
cal appearance, virtue, having God’s love, having love and support from family, 
outdoing others in competition, and obtaining others’ approval [51]. A key predic- 
tion of the CSWs model is that the impact of life events on self-esteem and affect is 
proportionate to the relevance of such events to one’s contingency of self-worth. In 
other words, our self-esteem is more affected (positively or negatively) by events 
that occur in areas of life that are relevant for us—and for our identity. For example, 
students who strongly based their self-worth on academic competence experienced 
lower state self-esteem when they performed poorly on academic tasks, received 
lower-than-expected grades, or were rejected from graduate schools, compared to 
those whose self-worth was less based on this domain or did not experience self- 
threat [56, 57]. 


The Motivational Force of Self-Esteem 


It is clear, at this point, that self-esteem is a complex psychological variable and thus 
there is not a unique way to assess it. In order to effectively operationalize self- 
esteem, it is important to have clear what aspect of self-esteem is important to a 
given research question and be aware of the objective limits of the measures avail- 
able in the literature. 

Perhaps, one of the most worrisome issues related to the abovementioned mea- 
sures is their self-report nature, which is also a wider issue in psychological research. 
Self-report measures entangle several advantages but limitations too, such as mem- 
ory limitations, recall biases, and social desirability sensitiveness, as in the case of 
the SISE which was found susceptible to acquiescence and social desirability [31, 
35]. To overcome these measurement problems, some researchers have assessed 
implicit self-esteem (Implicit Association Test-IAT; [58]) by assessing automatic 
associations of self (through reaction times) with positive or negative valence [59]. 
However, the studies revealed unclear results, as construct divergence between 
implicit and explicit measures of self-esteem emerged. This result might seem con- 
tradictory, yet it highlights that the self-report measures of self-esteem developed to 
this point, although reliable, valid and all somehow different from each other, have 
all a common limitation. The operationalization of self-esteem so far developed 
takes into consideration its cognitive (evaluation of oneself, e.g., RSE), affective 
(emotional responses, e.g., SSES), behavioral (competence, e.g., SLSC-R), and 
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value components (domain important to the individual, e.g. CSWs), yet they neglect 
an important aspect of self-esteem: namely, its motivational component. 
Understanding self-esteem as a fundamental psychological need implies that people 
are motivated to gain and maintain high levels of self-esteem: in other words, self- 
esteem is a goal in and of itself [60]. According to this conceptualization, self- 
esteem is a self-motive: it provides both a standard and a direction for behavior. 

However, although the idea that this motive underlies human behavior has been 
a central theme in psychological theorizing (e.g., [33, 37, 61—65]), self-esteem has 
been traditionally assessed as a status (situational or stable) that characterizes the 
individual and does not capture the desire for a high self-esteem. In defining self- 
esteem, Crocker and Wolfe [33] recognize the motivational component of such psy- 
chological construct; yet, their operationalization of self-esteem does not capture 
whether the individual is striving to increase the level of self-esteem. Indeed, this 
operationalization of self-esteem as a motivational concept is, perhaps, the missing 
link to understand the behavioral strategies with which the individual might decide 
to engage to pursue or regain optimal levels of self-esteem. 

As mentioned, many scholars have indeed assumed that people possess a motive 
or need to maintain self-esteem (e.g., [27]). The abovementioned sociometer theory 
grasps very well this idea, by conceptualizing the motivational force of self-esteem 
(namely, the self-esteem motive) as the human impulse of minimizing the likeli- 
hood of relational devaluation, or in other words, of rejection [25, 26]. This, in turn, 
leads to the pursue and preservation of high self-esteem. In fact, people typically act 
in ways that they believe will be of help in increasing their social acceptance by 
increasing their relational value: when this process is achieved, the individual’s self- 
esteem will be enhanced. On the contrary, when a given situation, a behavior, or 
even an anticipated, potential, or irrational consequence of a behavior, might hinder 
the social or relational value of a person, her self-esteem is reduced or, at least, per- 
ceived to be threatened. Indeed, events that are known (or potentially known) to be 
“public”, and therefore are more socially laden, have great effects on self-esteem; 
rather, “private” events, that are lived (or perceived to be lived) only by the indi- 
vidual, are usually less influential on self-esteem: if self-esteem was determined 
exclusively by private self-judgments, socially laden events should have no greater 
impact on self-esteem than private ones [27]. 


The Dark Side of Self-Esteem 
Perhaps, one of the most significant and influential consequence of this conceptual- 
ization lies in the understanding of the crucial role of self-esteem in undesirable 
behaviors. Indeed, although self-esteem has been historically associated with desir- 
able outcomes (e.g., better performance; [6]), it also recognized that high self- 
esteem can be associated with certain dysfunctional psychological processes and 
undesirable behaviors, such as egotism, narcissism, and violence: the dark side of 
self-esteem [66]. 

Within this conceptualization, the defensive aspect of self-enhancement grasped 
by the MSEI gains significant relevance. As mentioned, this inventory differentiates 
between secure vs insecure self-esteem, with high scores expressing a biased 
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self-presentation which tends to always denies weaknesses and claims strengths. 
This biased self-presentation may correspond to a non-genuine self-evaluation [24]. 
Therefore, this non-authentic self-assessment (being conscious or not) might cover 
other negative patterns of self-evaluations, such as those related to negative, insta- 
ble, contingent or narcissistic self-appraisals. In this conceptualization, the defini- 
tion of self-esteem as a motive potentially could explain the heterogeneity of 
individuals with high self-esteem, encompassing people who honestly acknowledge 
their good (and bad) qualities along with narcissistic, defensive, and arrogant indi- 
viduals [6]. Simply put, the definition of self-esteem as a motive would presuppose 
that people would strive for high self-esteem, no matter if it is non-genuine or 
authentic. 

Indeed, high scores in overall self-esteem coupled with high variability in the 
worthiness dimensions of the MSEI are often associated with high scores of narcis- 
sism and aggression [67]. What follows, is that high and optimal self-esteem actu- 
ally are two distinct constructs: the former can be fragile or secure, while the latter 
is characterized by genuine, true, stable, and congruent (with implicit self-esteem) 
high self-esteem [67, 68]. Likewise, contingent self-esteem studies (e.g., [13, 69]) 
have also argued for the existence of the dark side of positive self-appraisal, in 
which individuals become psychologically vulnerable due to their dependence on 
external validation for their self-esteem. 

General principles from this work may be useful for clinical and non-clinical 
consideration alike. Yet, no specific implications have been developed or tested for 
assessing or intervening to address contingent self-esteem in a way that could con- 
sider positive authentic and dark self-esteem simultaneously, as well as its motiva- 
tional component. 


Future Directions in Assessing Self-Esteem 


As we described beforehand, researchers have developed several kinds of measures 
targeting different features of self-esteem. To sum up, the literature provides mea- 
sures to assess the state self-esteem, trait self-esteem, and domain-specific self- 
esteem. However, the quantitative psychological studies carried out to develop such 
measures relied heavily on surveys and laboratory experiments, which have well- 
known and long-endured limitations. In fact, on the one hand, surveys require peo- 
ple to make retrospective and often generalized judgments, which tend to be affected 
by memory limitations and recall biases [70, 71]. On the other hand, laboratory 
experiments do not take into account the context of a person's daily life which can 
influence her states and responses [72, 73]. Taken together, these considerations 
raise questions about the ecological validity of theoretical and methodological con- 
clusions if not coupled with field data [74]. 

Furthermore, it seems that among several possible theorizations and measures of 
self-esteem, there is still a lack of consensus upon the most adequate ones. Obviously, 
it would be too simplistic and trivial to presuppose a priori which theoretical model 
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and measure should be considered the finest to grasp the most precise definition and 
operationalization of self-esteem. We suggest that the best possible way to over- 
come such limitations—at least in part—and therefore to select the most efficient 
and accurate measure of self-esteem, is threefold: (a) to consider carefully the 
aspects of the psychological process under scrutiny; (b) to understand which facets 
of self-esteem would be relevant to the main research question; and (c) to take into 
account which scientific method (e.g., correlational, experimental, etc.) will be most 
suitable and effective in a given setting. 

Nevertheless, as Baumeister et al. [75] argued, more attention should be given to 
how individuals actually behave in various situations rather than rely on what they 
claim, recall having done, or believe they would do in hypothetical situations. In 
addition, more attention should be given to the interaction between individual and 
social idiosyncrasies of self-esteem [26], as a growing amount of evidence shows 
that different constructs related to the self should indeed incorporate individual, 
social and cultural components (e.g., [76]). 

Within this perspective, the widespread use of mobile technologies opens up new 
opportunities of collecting (social-) psychological data in specific contexts and situ- 
ations. For example, mobile technology is already widely used to optimize health 
behavior change interventions (e.g., [77]). More specifically, the use of new digital 
technologies can help to overcome the limitations of the existing measures of self- 
esteem, giving the opportunity of looking at changes in self-esteem that occur in a 
field setting and therefore within the daily life moments. Such new data could not 
only be relevant to the understanding of self-esteem per sé, but could be coupled 
with other types of data collected through various types of sensors (e.g., global 
positioning systems-GPS, microphones, cameras, activity and sleep monitors, heart 
rate monitors, etc.) or software (e.g., social networks activity, mobile apps, etc.). 
Therefore, thanks to the means provided by new digital and wearables technologies, 
for the first time self-reported measures of self-esteem can be combined (above and 
beyond other psychological measures) to physiological and behavioral measures. 

Along this line of research, Quantified Self-enabled data collection procedures 
could be advantageous. The quantification of the self (Quantified Self—QS) is a 
form of self-tracking an one’s own daily life activities and behaviors, which eventu- 
ally allow for the analysis of behavioral trends and patterns across a variety of life 
domains (e.g., physical activity, nutrition, weight; [78]). Generally, QS helps the 
individual to reflect upon such patterns and trends, and potentially leads to the appli- 
cation of behavior change strategies built upon these data. Theoretically, the process 
of self-monitoring and self-tracking one’s own data could be useful to kick-start a 
process of behavior change across different life domains by enabling the person to 
change her relation to her own body and health, and to better control health-related 
decisions [79]. However, although some evidence seems to corroborate these 
hypotheses (e.g., [80, 81]), caution should be used in implementing interventions 
based on QS. First, recent research shows that an excessive quantification of one’s 
own actions and behaviors could lead to psychological distress [82, 83]. Second, 
and perhaps most importantly, an enormous amount of research shows that in order 
to promote long-lasting behavior change one will need to detach from the 
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monitoring of external objects and data, and rather should “monitor” his/her own 
internal physiological and psychological states and processes: in fact, by focusing 
on internal rather than external rewards and therefore by increasing intrinsic motiva- 
tion in the new behavior (e.g., [84]), the person will in turn develop to greater self- 
awareness, presence in the moment, values and meanings, and perhaps identity 
change in the long term [85-87]. 


New Technologies and Social-Psychological Processes 
and Constructs 


However, as mentioned before, the opportunities arising from the digital era 
should not be overlooked. In the last couple of decades, plenty of research has 
shown the link between various technological tools (both hardware and software) 
and a plethora of psychological processes and characteristics. Since the spread of 
mobile phones and smartphone, one track of research aimed to study the relation- 
ship between the (mis)use of technology and psychological processes (e.g., [88— 
90]). For instance, Andreassen et al. [88] examined the effect of personality traits 
such as narcissism, socio-demographic characteristics, and self-esteem on addic- 
tive social media. Among the others, and accordingly to previous research [91— 
93], they found that positive self-concept (i.e., high self-esteem) was negatively 
related to social media addiction, indicating that the Internet may provide a differ- 
ent social arena from the in the face-to-face life to enhance self-esteem. This track 
of research is extremely useful and much needed, since it helps clarifying (and 
will continue to do so) the psychological processes which can cause, be related, or 
be caused by the functional or dysfunctional use of different technology-based 
tools. For example, personality traits can predict Facebook use [94], while tech- 
nology-based self-help therapies—which relies on minimal contact with an actual 
therapist—have proven effective, low-cost interventions at least for anxiety and 
mood disorders [95]. 

Furthermore, together with the development of social media, wearable devices, 
machine learning, big data, data mining technologies, internet of things, etc., another 
track of research has been consistently growing and aims to leverage these new 
technologies and related digital tools in various healthcare-related domains [96, 97]. 
More specifically within the social-psychological domain, this track of research is 
extremely interesting since it could unveil a totally new research filed which will 
transform the way we measure social-psychological constructs and processes: it is 
indeed possible to leverage new technologies to measure such constructs and pro- 
cesses—which have been historically assessed by self-reported measures exclu- 
sively—through behavioral data collected in a unobtrusively and longitudinally 
manner (via wearables, smartphone use, smart home and smart office, smart meters, 
etc.; e.g., [98]). Some research has already succeeded in this goal. For example, 
aggregated smartphone usage features can predict the Big-Five personality traits 
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[99]; Facebook likes can predict, among other sensitive personal attributes, person- 
ality traits, intelligence, happiness and addictions [100]; and, the combination of 
mobile phone usage and sensor data can predict with 75% accuracy low and high 
perceived stress [101]. It is therefore clear that the relationship between automati- 
cally extracted physiological and behavioral characteristics derived from rich data, 
and self-reported measures of social-psychological constructs and processes, could 
be disentangled eventually, and could potentially lead to new, behavior-based, eco- 
logically valid measures of such facets. For example, Sun et al. [102], correlated 
gait patterns with self-reported self-esteem measured via the RSS: first, all partici- 
pants completed the RSS; then they walked for 2 min on a rectangular carpet, while 
their gait data were recorded using a Kinect sensor. Based on machine learning, the 
authors were able to build predicting models to recognize self-esteem, with signifi- 
cant results (r = 0.45 for males; r = 0.59 for females; both p < 0.001.) 


Toward New Methods of Quantifying and Conceptualizing 
Self-Esteem 


Based on the abovementioned considerations, and specifically in relation to self- 
esteem, we propose a new approach to measuring such construct which could shed 
light, on the one side, on its theoretical understanding and definition and, on the 
other side, on its applied implications and applications to promote greater psycho- 
logical wellbeing and to enhance quality of life. Drawing upon recent developments 
in applied social psychology (e.g., [98]) the suggested approach would combine 
both self-reported and digital technology-based tools. Specifically in the context of 
Quality of Life Technologies (QoLT)—technologies for assessment or improve- 
ment of a person’s quality of life [103]—we believe that the contingent measure- 
ment of self-esteem via self-report and digital technology-based tools, could 
potentially shed light on its facets and processes above and beyond what has been 
understood so far through the standalone implementation of self-report measures. In 
turn, this could potentially expand the goals of QoLT above and beyond its already 
defined aims [103]: QoLT could aim at clarifying and expanding the theoretical 
understating of latent variables included in the WHO definition of QoL, such as 
self-esteem. 

In our view, the combination of self-reported and digital technology-based tools 
would allow researchers and practitioners to take into account the two missing fac- 
tors that seem important to assess self-esteem, namely, the momentary experience 
and the social context. With reference to the former, we suggest that future research 
should investigate via digital tools the momentary experience of self-esteem, namely 
its daily fluctuations. Indeed, past research has shown that self-esteem: (a) is most 
closely connected to a specific class of emotions—rather than to the whole spec- 
trum of emotions—that relate to how people feel about themselves in a specific 
moment [12]; (b) that self-esteem indeed is open to momentary changes, raising 
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and falling as a function of one’s aspirations and failure/success experiences [36]; 
and that (c) one of the simplest way to measure state self-esteem is directly to ask 
individuals how they feel about themselves at that given moment via a single-item, 
valid and reliable measure [26]. In this vein, new technologies may be used to 
monitor the momentary experience of self-esteem, through collecting visual data 
(e.g., posture, clothes worn, facial expression and or posture in 'selfies' etc.), audio 
data (e.g., voice volume/pitch, etc.), text data for content analysis (e.g., text, social 
media posts, etc.), emoticon usage for sentimental analysis (e.g., number/type/of 
emoticon). 

With reference to the latter, the social context, we believe that longitudinal infor- 
mation derived from new technologies usage (such as behavior shown on social 
media—likes, comments, selfies posted, etc.; data gathered from GPS enabled 
devices; physiological data such as heart rate variability, etc.), if combined with 
validated measures of self-esteem, can be indeed used to indirectly measure both 
the affective (e.g., self-liking) and behavioral (e.g., competence) components of 
self-esteem and to couple such components to the specific social context in which 
self-esteem components are assessed. In fact, previous research shows that: (a) self- 
esteem can be considered a monitor of social acceptance and therefore it motivates 
people to behave in ways that would make them valuable in specific social context 
which is perceived to be relevant by the individual [26, 27]; (b) the self-esteem 
motive, functioning as a tool to avoid social devaluation and rejection, should be 
measured in-context by contingent measures of such construct [51]; and (c) in- 
context measures of self-esteem can shed light on how self-esteem is implicated in 
affect, cognition, self-regulation of behavior and social processes, and can possibly 
unveil solutions to debates about the nature and functioning of self-esteem [34, 
69]—Aand, in turn, suggest how self-esteem is causally related to mental health, 
wellbeing, performance and quality of life [6]. 

According to the abovementioned evidence, we believe that it would be possi- 
ble to leverage new technologies specifically in two (or, arguably, one) methods of 
research: the Experience-Sampling Method [104] also known as the Ecological 
Momentary Assessment [74, 105]. Certain technology-based applications of such 
methods are already in use and show important mechanisms and processes that are 
relevant to the assessment and treatment of mental health—e.g., PsyMate™ [106, 
107]—, such as the use of machine learning in predicting therapeutic outcomes in 
depression [108]. Along the same line of research, initial evidence reveals the 
efficacy of passive sensors for predicting self-esteem: by using machine learning 
techniques, it has been possible to relate performance, social and appearance self- 
esteem to several mobile-based digital behavior categories (such as calls, texts, 
conversations, and physical activity; [109]); furthermore, preliminary data indi- 
cate that is could be possible to correlate gait data to self-esteem with a fairly good 
criterion validity [102], suggesting that posture and perhaps other body-language 
characteristics could be taken as a good supplementary method to measure 
self-esteem. 
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Conclusion 


In conclusion, the present chapter shows that far more research is needed to really 
uncover the potential of a variety of digital and technological tools in the broader 
field of applied psychology—such as mobile and wearable technologies used in dif- 
ferent contexts of applications for the promotion of better quality of life [110—112]. 
Within this realm, new measures and conceptualizations of self-esteem should 
likely depend on both intrapersonal and interpersonal/social factors. We suggest 
that academics and practitioners alike should try to take advantage of the opportuni- 
ties provided by the digital age in trying to further understand and measure the 
concept of self-esteem, by specifically capturing the level of self-esteem in context 
and anchoring such level to specific behaviors. Such new methods might make both 
the intrapersonal and social influence of self-esteem salient and decipherable. 
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Chapter 12 A) 
Yoga and Meditation for Self-Empowered ss 
Behavior and Quality of Life 


Gerlinde Kristahn 


Do not believe in anything simply because you have heard it. 
Do not believe in anything simply because it is spoken and 
rumored by many. Do not believe in anything simply because it 
is found written in your religious books. Do not believe in 
anything merely on the authority of your teachers and elders. 
Do not believe in traditions because they have been handed 
down for many generations. But after observation and analysis, 
when you find that anything agrees with reason and is 
conducive to the good and benefit of one and all, then accept it 
and live up to it. 


Buddha 


Introduction 


Quality of life (QoL) is defined for the purposes of this chapter along the World 
Health Organization (WHO) definitions as an individual's physical and psychologi- 
cal health, their social relationships and the environment in which they live [1]. 
According to the criteria used in the Scales of General Well-Being (SGWB) ques- 
tionnaire, the main contributing factors to an individual's QoL are well-being crite- 
ria such as happiness, vitality, calmness, optimism, involvement, self-awareness, 
self-acceptance, self-worth, competence, development, purpose, significance, self- 
congruence and connection [2]. The importance of an individual's spirituality and 
personal beliefs for their well-being, QoL and health status have been largely 


! (This citation has been taken from the website Knowing Buddha: https://www.knowingbuddha. 
org/unique-knowledge) 
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confirmed through numerous scientific studies [3, 4]. Personal beliefs are what an 
individual believes to be the truth, or the beliefs that define their individual world- 
view [5]. Additionally, individuals have a tendency to believe that others “feel, think 
and act" as they do [5]. Personal beliefs [6] are crucially linked to individual well- 
being and the individual’s “engagement to explore—and deeply and meaningfully 
connect one’s inner self—to the known world and beyond.” [7] This definition 
focuses on a personal search for meaning and exploration of QoL [7]. The definition 
provided by WHO the focuses on the “person’s personal beliefs and how these 
affect quality of life". 

Besides personal beliefs and their connection to a good QoL and happiness, this 
chapter also explores possibilities regarding self-empowered behavior change. Self- 
empowerment describes the way individuals “promote beliefs and attitudes favor- 
able to deferring immediate reward for more substantial future benefit”? [8] We 
consider behavior mainly as habit, or as an individual’s unconscious acts; therefore, 
producing a change in behavior involves repeated adjustment, as “changing behav- 
iors is not about changing one act; it is about altering the routines in which the acts 
are embedded.” [9] 

Another central component of personal-level QoL is the notion of happiness. 
Research has shown happiness to be a key aim for individuals [10, 11]. It has been 
widely shown that happiness plays a crucial role in the search for the meaning of life 
in religion and spiritual understanding, as well as in one’s personal beliefs (see also 
below). Numerous scientific studies have explored what makes individuals happy 
and the main factors for experiencing bliss, self-realization and QoL. However, 
there are important differences between how to be happy and what makes individu- 
als happy [12]. In this chapter, happiness is defined as a constant positive feeling or 
subjective attitude towards one’s life [10—12] and is viewed on a continuum from 
happy to unhappy. This definition differs from others that define happiness as spe- 
cial moments where individuals feel bliss [10]. 

Happiness, as an essential component of QoL and well-being, does not only 
depend on the environment in which one lives. Individuals with enormous wealth 
and comfortable living conditions can be depressed, whereas others living in adverse 
conditions can manage their life circumstances to achieve a high degree of inner 
happiness [13-18]. Objective variables (e.g., income or lack of traumatic experi- 
ences) are far less significant determiners of happiness than intuition and the way 
one views their life experience [13-17, 19]. Indeed, social pressure and stress can 
occur when humans primarily consider happiness as a life achievement and 


>To learn more: 

“There are four factors which Tones (1993) considers central to the concept of empowered 
action for the individual: (1) The environmental circumstances which may either facilitate the 
exercise of control or, conversely, present a barrier to free action. (2) The extent to which individu- 
als actually possess competencies and skills which enable them to control some aspects of their 
lives, and perhaps overcome environmental barriers. (3) The extent to which individuals believe 
themselves to be in control. (4) Various emotional states or traits which typically accompany dif- 
ferent beliefs about control—such as feelings of helplessness and depression, or feelings of self- 
worth.” p. 1274. 
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unhappiness as a failure [20-22]. What is less clear, however, is how striving for 
happiness influences human belief systems [23-27]. 

In this chapter, we explore the role of thoughts, personal beliefs and emotions in 
the development of behavioral habits. Considering stress as an example of a factor 
influencing QoL that is shaped by one's habits, we propose yoga and meditation as 
practices that can help individuals achieve self-empowerment by altering their 
thoughts, beliefs, emotions and, ultimately, their behaviors, thus resulting in a 
reduction of stress and an improvement in QoL. Furthermore, we discuss the poten- 
tial use of technological devices and other tools for quantifying thoughts, beliefs, 
emotions and behaviors as a means of further empowering behavioral change. 

The chapter is structured as follows. First, we provide a short overview of some 
relevant concepts such as the influence of stress and emotion on the human body 
and mind (Section "Stress and Emotions' Influence on Psychophysiological 
Factors"). Second, adopting an interdisciplinary perspective, we explore how human 
behavior, and therefore QoL, is influenced by the mind (thoughts), by one's belief 
in the thought and by the emotions produced by one's thoughts (Section "Thoughts, 
Emotions, Beliefs and Behaviors"). Next, the chapter offers a discussion that can 
contribute to readers' understanding of how stress-relieving techniques such as 
yoga and meditation can improve QoL by changing an individual's thinking, emo- 
tions, beliefs and related behaviors (Section "Behavior: Yoga and Meditation as 
Interventions for Stress Management"). Specifically, yoga and meditation are 
explored as techniques to (1) observe thoughts (the mind), (2) separate beliefs from 
thoughts (personal beliefs) and (3) understand the connections that lead from a 
belief to a feeling (emotion) and (4) from a conscious or unconscious feeling to a 
behavior (behavior). Based on several case studies, we then suggest new technology- 
enabled ways of quantifying thought, belief, emotion and behavior that can be used 
by practitioners of yoga and meditation to assess the impact of their practice on 
these factors that influence QoL (Section “Quantifying Thoughts, Beliefs, Feelings 
and Behaviors"). The possibilities of self-empowered behavior change through 
yoga and meditation, as well as psychophysiological assessment tools, are discussed 
in section “Yoga and Meditation as Techniques for Achieving Self-Empowered 
Behavior". Finally, a conclusion to the chapter is provided in section “Concluding 
Remarks". 


Stress and Emotions" Influence 
on Psychophysiological Factors 


Stress can be defined as “the non-specific response of the body to any demand made 
upon it." [28] Individuals consider all subjective experiences to be either pleasant or 
unpleasant to some extent. Therefore, an individual's subjective experience can 
have a very positive or very negative impact on their overall well-being, health and 
QoL [29]. The mechanisms of stressful thoughts leading to disease [28] are 
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Table 12.1 Most common categories of emotions [36] 


Emotions without 
clear Valence 
Negative Emotions Positive Emotions Connotation 
(a) anger (approach-oriented anger, (a) affection (love, tenderness, | (a) surprise 
withdrawal-oriented anger, anger sympathy) (wonder) 
in defense of other, anger in (b) amusement (humor, mirth, | (b) suspense 
self-defense, indignation) happiness in response to 
(b) anxiety (dental anxiety, slapstick comedy) 
performance anxiety, agitation) (c) contentment (pleasure, 
(c) disgust (disease-related disgust, serenity, calmness, 
food-related disgust) peacefulness, relaxation) 
(d) embarrassment (social anxiety, (d) happiness (except 
shame, social rejection) happiness in response to 
(e) fear (threat) slapstick comedy) 
(f) sadness (achievement failure, (e) joy (elation) 
dejection, depression) (f) anticipatory pleasure 
(appetite, sexual arousal) 
(g) pride 
(h) relief (safety) 


becoming an increasingly important field of scientific research. Studies have shown 
that an individual’s attitude towards stress plays a crucial role in their health and 
well-being. However, the absence of stress in one’s life can have an equally negative 
influence on one’s health as can an excess of stress, since only an adequate balance 
between the two generally leads to greater well-being [30]. 

Stress itself is not an emotion, although it is often created by certain feelings 
[31-33]. Emotions “typically unfold dynamically” [32] and “have a beginning and 
an end,” [32] whereas feelings are present for a longer period of time [32]. Humans 
usually distinguish positive and negative feelings as those that create emotions and 
impact the “fluidity or strain of life process(es).” [34] Often, feelings are considered 
to lead to negative emotions when they throw life out of balance and cause struggle. 
Feelings are regarded as producing positive emotions when life is harmonious, effi- 
cient and flowing [34]. Familiar patterns often allow life to continue smoothly, 
whereas changing patterns create an unfamiliar situation that “generates feelings 
and emotions.” [35] The emotions that have been evaluated most often in the psy- 
chological studies are presented in Table 12.1 [36], with each one categorized 
according to its valence (positive, negative or unassigned). [37] 

Although there are numerous studies that have demonstrated the difficulties 
involved in relating emotions to psychophysiological reactions in the human body 


3Emotional valence (Definition). 

“The concept of emotional valence, or simply valence, has been used among emotion research- 
ers to refer to the positive and negative character of emotion or some of its aspects, including elici- 
tors (events, objects), subjective experiences (feeling, affect) and expressive behaviors (facial, 
bodily, verbal). The valence of feelings has been argued to be a pivotal criterion for demarcating 
emotion and a core dimension of the subjective experience of moods and emotions.” 
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[36], there are examples from the literature that provide persuasive evidence regard- 
ing particular correspondences between physiology and emotional states. In partic- 
ular, the studies in the latter group indicate that, on the physical level, the body 
reacts to any situation in its environment that creates psychological flow or strain. 
For example, if one part of the body is changed or influenced by a situation, as in the 
case where one consciously slows down one’s breath, the other body parts (such as 
one’s heart rate) adapt to the new situation. This reaction is indicative of “synchro- 
nized interactions among multiple systems,” [38] other examples of which, relating 
to heart rate, are shown in Fig. 12.1 and explained further below. 

Overall, common repetitive behaviors, such as inactivity, poor nutritional habits 
and alcohol and tobacco consumption, may affect the way an individual manages 
daily life stress [40] and can determine up to 40% of the individual’s future health 
state and life quality [41]. Meanwhile, there are various psychophysiological factors 
that can be signs of stress (e.g., galvanic skin response, respiration, blood pressure 
and heart rate) [36]. In this section, we have chosen to discuss heart rate as a main 
indicator of stress, as it has been demonstrated that one’s heart rate responds directly 
to momentary emotional states [42]. While an abnormally low 24-hour heart rate 
can indicate increased risk of heart disease and premature mortality, it is also a 
strong marker of stress and emotions [43], as is shown in Fig. 12.1, and thus dem- 
onstrates the “synchronized interactions between different systems in the body” 
[44] alluded to above. It can be observed that heart rate patterns in particular—that 
is, the patterns of the curves rather than specific heart rate values—often directly 
reflect and synchronize with an individuals’ emotional states [44]. 

Furthermore, the analysis of heart rate and heart rate pattern provides an indica- 
tor of neurocardiac fitness and autonomic nervous system function [42]. Some stud- 
ies have even found that heart rate pattern “covaried with emotions in real time.” 
[44] As heart rate patterns often directly reflect emotional states, various techniques 
to reduce stress, such as meditation techniques, produce specific heart rate patterns; 


Heart Rhythms (Heart Rate Variability) 
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Fig. 12.1 Example of Emotions reflected in Heart Rhythm Patterns [39] 
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in particular, such techniques have been found to produce a coherent, fine, smooth, 
“wave-like-pattern in the heart rhythms.” [45] Scientists have also found that a per- 
son practicing a technique to maintain positive emotional feelings displays heart 
rhythms that are synchronized with their brainwaves influenced by their emotional 
states (see Figs. 12.2 and 12.3). In comparison with individuals who experience a 
high degree of stress, those who apply such techniques also exhibit more regular 
and stable heart rate patterns. 

Recent research has revealed that the body is also able to develop coherence or 
synchronization with the physiological processes of others. For instance, a mother 
thinking about her in utero baby can lead to the synchronization of her brainwaves 
with her baby’s heartbeat [48]. It has also been shown that couples that have lived 
together for many years can have coherent heart rate patterns while they sleep [49]. 

To summarize this section, we observe that there is a synchronization between 
brain and cardiac activities [50, 51]. There is particular evidence of a relation 
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Fig. 12.2 Image of Dissonant Heart and Brainwave Rhythm [46] 
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Fig. 12.3 Image of Heart Rhythm and Brainwaves in sync during a Meditative State [47] 
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between heart rate patterns and positive and negative emotions [42]. Additionally, 
body stress regulation techniques enable a more regular and stable heart rate 
pattern.* 


Thoughts, Beliefs, Emotions, and Behaviors 


Behavior can be defined as “the internally coordinated responses (actions or inac- 
tions) of whole living organisms (individuals or groups) to internal and/or external 
stimuli.” [54] As indicated in the conceptual model presented in Fig. 12.4, behavior 
is influenced by (a) thoughts, (b) belief in thoughts and (c) feelings affected by 
beliefs and (d) decisions motivated by feelings [55]. Behavior can be influenced by 
“individual-level attributes as well as by the conditions under which people live.” 
[62] Thus, human behavior depends on the context in which it takes place [63]. 
Additionally, some studies state that as much as 99.9% of the decisions made by 
humans are shaped by other people [64]. Bargh provides further evidence of envi- 
ronmental influences on human behavior, stating that “our psychological reactions 
from moment to moment... are 99.44% automatic." [65] Therefore, in order to bet- 
ter understand how an individual makes decisions, one needs to observe them within 
their natural environment [66] and consider contextual factors, as is indicated in 
Fig. 12.4. This principle must be applied when one is assessing the influence of 
human thinking on individual belief systems and emotions, leading to behaviors. 

As mentioned above, this chapter considers behavior in relation to stress- 
management. In section “Stress and Emotions’ Influence on Psychophysiological 
Factors", it was seen that emotions and feelings represent an important factor in 
stress management. The model presented in Fig. 12.4? conceptualizes the steps by 
which an individual adopts a behavior that can either increase or reduce stress, 
beginning with a change in one's thoughts, beliefs or feelings. 


^ Recent studies found out that individuals can influence and measure these impacts on emotions, 
for example psychophysiological functions like the heart rate. According to the HeartMath- 
Institute, "These positive emotion-focused techniques help individuals learn to self-generate and 
sustain a beneficial functional mode known as psychophysiological coherence, characterized by 
increased emotional stability and by increased synchronization and harmony in the functioning of 
physiological systems." [43]. This tool helps restructuring emotions, and takes usually 5-15 min. 
For in-depth descriptions of these techniques, see [52, 53]. 
> Disclaimer: the illustration visualizes what has been described hereafter, but is not yet empirically 


validated. Individual & Contextual Factors of Behavior (own graph based on empirical confirmed 
findings from): [55-61]. 
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Fig. 12.4 Individual and contextual factors of behavior 


The Influence of the Mind on Human Behavior 


Corresponding to the first circle (Thought) in Fig. 12.4, thoughts are related to 
“one’s own individual experience.” [67] Most psychological research on decision- 
making over the past 30 years has focused on the logic of rational, controlled deci- 
sions (i.e., the logic behind behaviors). While logical mechanisms may describe the 
decision-making processes of scientific pursuits, their analysis is less appropriate 
for understanding the uncontrolled aspects of humans' ordinary changes in behav- 
ior, which are based on both thoughts and feelings [56]. According to Baumeister 
et al., “it is plausible that almost every human behavior comes from a mixture of 
conscious and unconscious processing." [60] This also means that a description of 
one's conscious thoughts alone (or unconscious thoughts alone) is rarely sufficient 
to explain a specific behavior. 

The mind, defined for the purposes of this chapter as an individual's collected 
thoughts and cognition, is one of the great influencers of behavior [68]. According 
to scientists such as Anderson, if someone envisions an action, it leads to an 
"increased intention to do it" [69] and therefore a greater chance that the action will 
be achieved. According to Tiller, meanwhile, individuals can influence their future 
with their thoughts in the form of intentions [57]. The systems of thinking, or per- 
sonal beliefs, that affect one's intentions begin to be developed in one's youth 
through imitation of parents, relatives, teachers and figures in media, among others. 
The information one gains in the process serves to “structure that person's under- 
stood world and purposive ways of coping in it.’ [70] When an individual is 
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confronted with a choice, they unconsciously call upon all the past experiences that 
have been collected and registered in their brain, recalling them as thoughts. This 
process has been widely studied in psychology as the “theory of planned behavior.” 
[71, 72] 

The effect of one's state of mind, including one's thoughts, on the outcome of a 
task has been demonstrated in a study by Armor and Taylor. In this study, the authors 
assessed how implemental and deliberative mindsets—the former characterized by 
plans and thoughts regarding how to implement an action, and the latter by ques- 
tions concerning the merits of the action—respectively influenced one's perfor- 
mance of a task. As the authors observed, “it appears that the simple difference in 
perspective that mindsets create—from the uncertain query ‘Will I do X?’ to the 
agentic assertion ‘I will do X’—can yield substantial differences in how the consid- 
ered action will be perceived and, subsequently, acted on.” [73] The authors con- 
cluded that the expected results of a task, as well as one’s predictions regarding 
one’s own performance, “become less favorable following deliberation and more 
favorable following thoughts of implementation.” [73, 74] 

Additionally, results from various studies have indicated that individuals adopt- 
ing the perspective of another, as opposed to simply discussing a situation from their 
own perspective, can be a successful strategy for producing behavioral change 
[75-77]. 

In summary, the mind significantly influences behavior across multiple dimen- 
sions. As is seen above, the mind retains past experiences that may be triggered by 
external impulses. These experiences consciously and unconsciously influence the 
ways individuals think, shaping their beliefs, emotions and actions. In the next sub- 
section, we will explore how beliefs (and unbelieves) can influence human behavior. 


Influence of Personal Beliefs on Human Behavior 


The second circle of Fig. 12.4, corresponding to beliefs, represents a step in the 
process leading to behavior in which an individual evaluates whether they believe a 
thought or not. A belief is not necessarily true; it is simply what an individual 
believes and claims to be true or false or right or wrong. It is also important to note 
that not all an individuals’ beliefs are derived from their own experience or thinking. 
As Barth et al. observes, “we all extend the reach and scope of our knowledge 
immensely, relying on judgements based on whatever criteria of validity we 
embrace—above all, what others whom we trust tell us they believe.” [59] 

What an individual hears and considers to be true will have a profound and long- 
lasting influence on their behavior. As Geraerts et al. have demonstrated, even “the 
false suggestion of a childhood event can lead to persistent false beliefs that have 
lasting behavioral consequences." [58] To illustrate this fact, the authors conducted 
a study where they informed subjects that they had become ill during childhood 
after eating egg salad. During the period immediately after receiving this false 
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information, and even four months later, these subjects tended to avoid eating egg 
salad, whereas the control group continued to eat it as usual [58]. 

The personal belief system that leads to such behaviors is useful, and in some 
cases it can serve to protect one from harm or save one’s life. This is especially 
important in dangerous situations, where one needs to quickly judge and react, often 
as part of a fight or flight response [78]. However, making quick judgements of this 
sort in a broader context may become problematic when an individual does not 
question their thoughts and beliefs and assumes that their thinking always captures 
the truth. These individuals might cling to their beliefs because of the familiar feel- 
ing that one experiences when the mind recognizes a situation [59]. In such cases, 
these feelings unconsciously lead the individual to make decisions based on their 
previously stored knowledge or beliefs, or “what a person employs to interpret and 
act on the world.” [59] However, if such an individual explores the ways they can 
alter their personal belief system and its impact on their behavior [79], they can 
ultimately achieve better QoL, especially in areas—such as physical activity, nutri- 
tion and sleep—that contribute to their health later in life [80-83]. 

An important example of personal beliefs includes stereotypes about the differ- 
ences between different cultural, ethnic and religious groups. Such beliefs, which 
derive from one’s one cultural, religious or social context, are often challenged 
when individuals from stereotyped groups behave in ways that are unexpected. The 
practice of sharing beliefs between members of different cultural, ethnic and reli- 
gious groups can lead to confidence-creating emotions which enable individuals to 
easily create relationships, as well as a willingness to cooperate with others [84, 85]. 

Spirituality, as a cultural concept, also significantly influences an individual’s 
beliefs [86]. Scientific research has attempted to measure spiritual well-being using 
scales, the most common of which are the Spirituality Index of Well-Being, the 
Spirituality Scale, and the Daily Spiritual Experience Scale. These scales assess 
spirituality using elements which ask the individual about their beliefs." 

Similar to spirituality, relaxation methods aim to help individuals manage stress 
through practices that affect one's personal beliefs. Many proposed methods for 
reaching a stress-free mental state include guidelines that instruct readers in chang- 
ing their behavior. These guidelines are often meant to encourage hopeful, optimistic 
beliefs and feelings of inner peace [87]. Some methods recommend practices such 


*The Spirituality Index of Well-being (SIWB) states that there is currently no instrument that can 
measure spirituality in its complexity, but it can deliver “health-related quality of life measures" in 
a qualitative form. The Spirituality Scale (SS) is a holistic approach measuring beliefs, intuitions, 
life choices and rituals. It identifies mainly three factors of spirituality: Self-Discovery, 
Relationships and Eco-Awareness. The Daily Spiritual Experience Scale (DSES) addresses spiri- 
tual experiences like awe, joy and a sense of deep inner peace. With this scale, emotions, cognition 
and behavior may be identified. According to the DSES, daily spiritual experience is related to 
improved Quality of Life and positive psychosocial status. As stated in the SIWB study, "Life 
scheme is similar to the construct of sense of coherence, which was described by Antonovsky as a 
positive, pervasive way of viewing the world, and one's life in it, lending elements of comprehen- 
sibility, manageability, and meaningfulness." 


7TFor example: “I accept others even if they do things I think are wrong" (Spirituality Scale (SS)) 
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as sitting down every morning to meditate [88] or drinking a cup of lukewarm water 
before eating [89]. While adopting these behaviors may be effective for some indi- 
viduals, not everyone will be able to attain from them the change in stress-reduction 
or health-improvement that they desire. Various approaches and techniques of teach- 
ing have therefore been developed, allowing each individual to determine what 
works best for them and what is most appropriate in specific situations [90]. 

In the health context, meanwhile, it has been demonstrated that healing beliefs 
can play an important role in health outcomes. Healing beliefs are a conscious and 
mindful determination one makes to improve one’s health, or an expectation that 
one’s health and well-being are going to improve and that one will live a meaning- 
ful, productive life. Healing beliefs are a way of understanding and giving meaning 
to illness and suffering through the continued belief that the patient will achieve 
healing and well-being [91]. 

Additional studies in this area have found that health expectancy (one’s personal 
belief regarding future health outcomes) is an important part of healing processes, 
especially when there is a close relationship between the healer and the patient [92]. 
Studies have demonstrated that a healer’s capacity to self-heal tends to have a posi- 
tive influence on a patient’s health outcomes. As Schmidt notes, “practicing loving 
kindness and compassion toward oneself helps develop these qualities in our rela- 
tionship with others.” [93] The self-healing capacity may be observed by the patient 
unconsciously, stimulating the patient to use the same skills to heal themself. 

In conclusion, there are many studies that demonstrate the impact of personal 
beliefs on individual behavior. More important than the act of changing one’s mind 
(one’s thoughts) in order to bring about behavioral change is changing one’s belief 
of this thought. This occurs on both a conscious and an unconscious level. The next 
subsection explores the influence of emotions on behavior, especially of those emo- 
tions that arise from beliefs. 


The Influence of Emotions on Human Behavior 


As is shown in Fig. 12.4, believing a thought can create certain feelings in individu- 
als that ultimately influence behavior. Feelings and emotions are interdependent. As 
mentioned in section “Stress and Emotions’ Influence on Psychophysiological 
Factors”, feelings produce emotions and last for a longer time than emotions, which 
always have a beginning and an end. Emotions influence behavior because an indi- 
vidual is more likely to do something if it is pleasurable and not stressful or if the 
outcome feels positive [94]. On the other hand, research shows that “anticipated 
emotion, especially anticipated regret, has been shown to motivate people and 
change behavior.” [60] Both examples illustrate how varied emotions, from positive 
emotions such as love to negative emotions such as fear, affect human behavior and 
supply important impulses for actions. 

A wide range of theoretical models of behavior have been proposed that demon- 
strate how behaviors leading to long-term improvements in health and QoL, such as 
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exercise and nutritional practices, can be implemented and adopted [95]. According 
to these models, individuals have the ability to adopt an alternative perspective or 
state of mind by altering their thoughts and personal beliefs, ultimately influencing 
their own emotions and leading to new behaviors. According to a recent study, 
mood is a factor that has a major influence on how effectively individuals are able to 
improve their behavior. Moods, which are defined here as states or frames of mind, 
can last for a brief moment or be present during longer periods [32, 97]. Individuals 
in a negative mood have been found to exhibit less attention and concentration on 
tasks than those in a positive frame of mind and to ultimately deliver worse results. 
Individuals in a negative mood have also exhibited particular difficulty adapting to 
tasks again after being distracted by their own thoughts. These results demonstrate 
that those in a negative mood commit more behavioral lapses when engaged in a 
task, have “task-irrelevant thoughts” with more frequency and exhibit less motiva- 
tion towards attentional commitments than those in a positive mood [96]. Meanwhile, 
a positive mood has been found to help individuals adapt to a task after a lapse [96]. 

A similar study concerning the impact of emotions on behavior has confirmed 
that emotions have a major influence in this regard [98]. According to Avey et al., 
individuals in a positive mood were not only better able to control their own behav- 
ior, but were also able to spread positivity to those around them. Their results dem- 
onstrate that one individual's positive mood can even influence f.ex. employees in 
an organizational structure. More specifically, the individual's mood was found to 
influence their own psychological capital, especially in the areas of hope, efficacy, 
optimism, resilience and mindfulness, by facilitating relationships and positive 
emotions, which led in turn to changes in the behavior of the surrounding individu- 
als [99]. 

When describing the impact of emotions on behavior, it is important to also con- 
sider how the way one is received or treated in one's surroundings affects emotional 
responses and, ultimately, behavior. Positive and negative feedback from one's envi- 
ronment, for instance, have been found to affect an individual's emotions in ways 
that influence behaviors such as “counterproductive behavior, turnover intentions, 
citizenship, and affective commitment." [100] 

In summary, there is significant evidence regarding the influence of emotions on 
an individual's behavior. This influence can be seen in the effects of mood on task 
performance, the impact of positive and negative feedback on behavioral change 
and the capacity of individuals to spread positivity and alter the behaviors of others, 
examples that demonstrate the crucial role emotions play in behavior choice 
and change. 


8 Fifty-nine undergraduate students (25 males) participated in this experiment for course credit. 
The mean age of the sample was 21.7 (SD 2.0; range 18-25) years. Individuals were allocated to 
positive (n 20), negative (n 20), and neutral (n 19) mood induction conditions using a counterbal- 
anced design. 

From the study: [96]. 
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Behavior: Yoga and Meditation as Interventions 
for Stress Management 


Given the above discussions concerning the influence of the mind, belief of thoughts 
and emotions on behavior, this section presents behavior techniques, particularly 
those of yoga and meditation, that have been found in research to help reduce stress 
through changes to an individual’s thoughts, beliefs and emotions that lead to 
behavioral change [101]. Nowadays, yoga and meditation are practiced worldwide 
to reduce stress and achieve life satisfaction [102—104]. Yoga and meditation tech- 
niques are claimed to be especially effective at helping practitioners (1) observe 
their thoughts (thought), (2) disconnect beliefs from thoughts (belief), (3) under- 
stand the connection between personal beliefs and feelings (emotion) and (4) recog- 
nize how conscious or unconscious feelings lead to a behavior (behavior).? 

In this chapter, yoga is defined as “an ancient practice with origins in India, [...] 
incorporating] muscle stretching, muscle relaxing, balancing poses (asanas), 
breathing (pranayama), guided relaxation, and/or meditation (dhyana) to achieve 
harmony and spiritual growth." [105, 106] It is a culturally based tool that can 
become an integrated habit in one's life [107]. A key aspect of yoga and meditation 
is a concept of embodiment that is focused on self-creation, as "control of the self is 
a project internal to the person rather than imposed by others, and modern persons 
actively define themselves through lifestyle choices, rather than being passively 
defined by their membership in traditional groups and moral orders.” [108] 

Research on the mindful body, which is an approach to the body that treats men- 
tal and physiological processes as aspects of an integrated whole, has revealed sig- 
nificant insight into the effect of emotions, passions and conversational practices on 
behavioral outcomes. As yoga and meditation seek to alter behavior through delib- 
erate changes to one's emotions, such findings are highly relevant to the analysis of 
yoga presented in this chapter [109, 110]. Meanwhile, yoga psychology is a field of 
research and practice focusing on the use of yoga as a psychological tool that inte- 
grates behavioral and introspective approaches to facilitate an individual's growth 
[111]. Several studies have revealed benefits of practicing yoga for a certain period 
of time. On a psychological level, yoga has been shown to improve emotional func- 
tioning [112] and reduce negative emotions in breast cancer patients [113]. In addi- 
tion, studies of embodied cognition have demonstrated that yoga “may reduce stress 
by affecting the way individuals appraise stressors.” [114] One study has also found 
that regular practitioners of yoga react to negative emotional stimuli in such a way 
that it *does not have downstream effects on [their] later mood state." [115] 

On a physical level, yoga reduces stress by stimulating exchange and harmoniza- 
tion between the body and the mind. One of the parts of the human anatomy most 
responsible for communication between body parts is the nervous system, particu- 
larly the vagus nerve, which corresponds to the central energy channel called 


?Individual & Contextual Factors of Behavior (own combination based on empirical confirmed 
findings from): [55-61 ]. 
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“Sushumna Nadi” in yoga [116]. According to experts, the vagus nerve communi- 
cates information mainly from the heart, gut and other organs to the brain [117]. 
This nerve is stimulated in all yoga postures, or asanas [118, 119]. Yoga practices 
are based on a distinction between the two sides of the body: the left, called “Ida,” 
which is feminine and lunar, and the right, called “Pingala,” which is masculine and 
solar. The aim of yoga is to unite both sides within the Sushumna Nadi or vagus 
nerve. When the body has been trained to slow its processes using the breath (e.g., 
via yoga), the heart rate automatically slows down when the individual is at rest. 
Neurological signals are transmitted from the brain to the heart and other organs via 
the vagus nerve without any change in the breath being necessary [120]. 

As is defined above, meditation is part of yoga practice. Meditation is defined as 
“practices that self-regulate the body and mind, thereby affecting mental events by 
engaging a specific attentional set.” [121] Meditation, like yoga, is considered a 
health-improving tool, and has been linked to numerous health benefits, such as 
reductions in pain sensation, short-term increases in blood pressure and improve- 
ments to respiratory efficiency, fluid exchange, cardiovascular system capacity and 
synchronization of the cells [122]. In a study of the efficacy of meditation tech- 
niques for treating medical illnesses, Arias et al. reported that “the strongest evi- 
dence for efficacy was found for epilepsy, symptoms of the premenstrual syndrome 
and menopausal symptoms. Benefit was also demonstrated for mood and anxiety 
disorders, autoimmune illness, and emotional disturbance in neoplastic disease.” 
[123] As Ospina et al. have observed in their survey of meditation research, the 
effects of meditation include physiological changes such as the reduction of the 
heart rate, blood pressure and cholesterol. On the neurophysiological side, medita- 
tion was also found to increase verbal creativity [124]. 

The calm mental states reached by practicing mindful meditation have been 
shown to impact the brain and emotions. As Hayes and Davis note, “There is evi- 
dence that mindfulness helps develop effective emotion regulation in the brain.” 
[125-128] The calm mental states associated with meditation can facilitate coher- 
ence or synchronization between bodily functions.'° Coherence tends to produce a 
sense of peace of mind and emotional balance that increases overall well-being 
[122] and QoL. The coherence of different psychophysiological functions can be 
developed through training such that the body, without specific stimulation, later 
applies these trained techniques unconsciously [129]. A meditation practice that has 
provided particularly effective for the achievement of calm mental states is 
Transcendental Meditation. Research on this method has focused especially on its 
relationship to stress reduction, decreased depression, reduced anxiety and improved 
sleep quality [130-132]. Other studies, however, have shown that more desirable 
psychological states can be achieved with the practice of yoga paired with medita- 
tion, especially in cases of burn-out syndrome, depression and mental disorders 
[133, 134]. 


10 Coherence is defined as the "synchronization or entrainment between multiple waveforms. A 
constructive waveform produced by two or more waves that are phase- or frequency-locked". 
Resource: https://www.heartmath.org/research/science-of-the-heart/coherence/ 
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The Center for Healthy Minds, led by Professor Richard Davidson at the 
University of Wisconsin-Madison,!! has stated, after many years of research on the 
topic, that well-being is a set of skills that can be learned in a way similar to playing 
the guitar. According to the results of their research, regular meditation can rewire 
areas of the brain [135] so that, although negative experiences from daily life are 
recognized, they affect individuals’ brain health, thoughts, emotions and actions for 
a much shorter time than they do for individuals who are not trained to meditate. !? 

As indicated in the previous section, several studies [136] have demonstrated that 
positive emotions can positively influence human behavior, health and overall well- 
being [45]. A key factor in this process is the cultivation of coherence between 
emotional and bodily states. If an individual regularly practices a meditation tech- 
nique that draws attention to positive emotions (such as Loving-Kindness Meditation 
[137] or other techniques that are viewed as stimulating positive emotions (such as 
progressive muscle relaxation [136]), the body system establishes a habit. Research 
results indicate [138—140] that past experience "builds within us a set of familiar 
patterns, which are instantiated in the neural architecture" [141] These patterns 
manifest as habits that are automatically enacted, even when the person is not 
actively practicing any meditation technique. External stimuli are then managed by 
the brain as perceptions, which automatically give rise to habitual feelings and 
behaviors [141]. According to one study, “the system then strives to maintain [posi- 
tive emotional habits] automatically, thus rendering coherence a more readily acces- 
sible state during day-to-day activities, and even in the midst of stressful or 
challenging situations." [122] In frequent meditators, stress responses are often 
automatically reduced the moment stress is experienced [142]. Interestingly, not 
every meditation technique leads to coherence between different body functions. 
Techniques that concentrate on positive emotions, for example, have been found to 
be more successful at facilitating coherent states than those that focus on the 
mind and thoughts (such as concentrative meditation) [122]. Therefore, one may 
assume that integrating the emotions into one's meditation practices, rather than 
merely concentrating the mind on a given object, can result in more effective physi- 
ological and psychological functioning. 

The approach to changing behavior through the cultivation of positive emotions 
is also considered in the understanding of impacts of yoga and meditation. According 
to the self-development perspective on behavioral change, the only way to change 
one's behavior is from within oneself [143, 144]. Individuals may need help from 
outside sources, such as other humans or their external conditions, but managing 
one's behavioral changes is the responsibility of each individual. This is also 


1 https://centerhealthyminds.org/ 
P https://centerhealthyminds.org/science/overview (Accessed 02.06.2020) 


13 Concentrative meditation is the term sometimes used for a type of meditation in which the mind 
is focused entirely on one thought, object, sound or entity. The intention is to maintain the single- 
pointed concentration for the duration of the meditation. For an object of concentration, the yogi 
may use a sound or mantra, their breath or a physical object such as an icon or candle. https://www. 
yogapedia.com/definition/10427/concentrative-meditation 
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confirmed by the results of research on intrinsic behavior change and motivation. 
Self-determined motivation in particular, an attitude which yoga training aims to 
cultivate and draw on, has been found to have a direct effect on behavior [145]. 

Both yoga and meditation are promising tools that can influence well-being and 
QoL by changing one’s relation to one’s thoughts, personal beliefs, feelings and 
emotions and behavior (as conceptualized in Fig. 12.5). Results from a variety of 
studies have confirmed these benefits; nevertheless, the exact mechanism by which 
meditation affects QoL is still unclear. It is known that meditation influences cogni- 
tive processes; however, there are other factors that also influence well-being [146], 
such as “cultural differences, socioeconomic status, health, the quality of interper- 
sonal relations, and specific psychological processes.” [146] 

In summary, the current state of research indicates that yoga and meditation can 
be helpful techniques for individuals to observe their own thoughts and make sense 
of the beliefs their thoughts give rise to. Furthermore, yoga and meditation affect the 
emotions and one’s resulting behavior in a positive way by reducing the stress 
caused by an individual’s environment. 


Quantifying Thoughts, Beliefs, Feelings and Behaviors 


When yoga and meditation are practiced as techniques of self-empowerment, as 
will be discussed in section “Yoga and Meditation as Techniques for Achieving 
Self-Empowered Behavior", it can be beneficial for the practitioner to assess 
changes to their thoughts, beliefs, emotions and behaviors through reliable means. 
This section therefore discusses how each of these variables can be quantified lever- 
aging recent developments in personal, miniaturized technology such as smart- 
phones and wearables to evaluate and ultimately improve the effectiveness of one’s 
practice. The table (Table 12.2) presents methods of quantifying these concepts 


Fig. 12.5 The effects of 0.67 
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Table 12.2 Examples of methods for the quantification of thoughts, beliefs, 
behaviors via technologies 
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emotions and 


Beliefs 
Mind (personal 
Data source | Dimension (thoughts) beliefs) Emotions Behaviors 
Self-reported | Patient Reported | Overall Overall Overall Overall 
Outcomes thoughts: PRO | beliefs: PRO | emotional behaviors: 
(PRO) and/or Momentary Momentary states: PRO PRO 
Ecological thoughts: self-beliefs: Momentary Momentary 
Momentary EMA EMA emotional behaviors: 
Assessments states: EMA | EMA 
(EMA) 
Other/ (ObsRO, Overall Overall Overall Overall 
partner- PeerMA) perceived perceived perceived perceived 
reported thoughts: beliefs: emotional behaviors: 
ObsRO ObsRO states: ObsRO | ObsRO 
Momentary Momentary Momentary Momentary 
perceived perceived perceived perceived 
thoughts: beliefs: emotional behaviors: 
PeerMA PeerMA states: PeerMA 
PeerMA 
Technology | Wearable Galvanic Skin | GSR Body or Body/parts 
reported (Psycho- Response HR (note: may | body-part of body 
Outcome physiology, (GSR) be impossible | movements movements 
(TechRO) Movement) Heart Rate to disentangle | Body Body posture 
(HR) from thoughts) | postureGSR, | GSR 
(note: may be HR HR 
impossible to Body Respiratory 
disentangle temperature rate 
from thoughts) Respiratory Voice 
rate analysis 
Voice analysis 
Face analytics 
Posture 
analysis 
Smartphone Analytics of | Analytics of | Context Context 
Behavioral search engine | search engine |(where, with | (where, with 
Metrics use use whom) whom) 
Social net Social net Interaction Interaction 
activity voice | activity voice | with context | with context 
(analysis of (analysis of Interaction Interaction 
content, e.g., | content, e.g., | with phone with phone 
asking asking 
questions) questions) 


using various data sources that are categorized in the manner defined by Mayo et al. 
in the context of patient care [147]. In this case, the tools and methods are adapted 
for the use of any individual in or outside of care settings. The first data source that 
is considered is self-reported data that the individual provides on their own. This 
includes information on both internal, unobservable states and external, observable 
states that is collected via patient-reported outcome (PRO) surveys for a given recall 
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period (e.g., the last month) and in shorter ecological momentary assessments 
(EMAs) [148]. The second data source includes data reported by other individuals, 
such as a friend, family member or caregiver, on the individual in question’s per- 
ceived internal and observable states. This information is collected in the form of 
observer-reported outcome (ObsRO) surveys and/or peer ecological momentary 
assessments (PeerMAs) [149], the content of which may differ from the PRO sur- 
veys and EMAs depending on what is being reported upon (e.g., internal, unobserv- 
able states or external, observable states). The third data source includes 
technology-reported outcomes (TechRO) datasets collected via wearable devices 
that record aspects of the individual's psychophysiology and their movements and 
via smartphones that record information concerning the individual's context and 
their interaction with their context. 

At present, the measurement of thoughts, beliefs, emotions and behaviors using 
the tools mentioned in Table 12.2, especially the technology-reported datasets, is 
only a hypothetical proposal. Concrete experimentation is still needed to ensure the 
accuracy, timeliness and reproducibility of the assessments of these variables [36] 
and the overall quality of the data collected. That said, these methods of quantifica- 
tion indicate a possible set of tools practitioners of meditation, yoga and other 
behavioral techniques might use to measure the effectiveness of their practices on 
their behavior and the variables—namely thoughts, beliefs and emotions—that ulti- 
mately shape behavior. 


Yoga and Meditation as Techniques for Achieving 
Self-Empowered Behavior 


Based on the research presented in this chapter up to this point, this section assesses 
yoga and meditation as techniques of individual empowerment through which one 
might change one's health behaviors to live a happier life. In particular, this section 
focuses on the use of yoga and meditation techniques for stress management as a 
means of enacting behavioral change. Self-empowered changing of one's health 
behaviors also implies an attempt to change one's perspective towards health and 
illness. Such a change might involve adopting the view held by Antonovsky, who 
stated, ^We are coming to understand health not as the absence of disease, but rather 
as the process by which individuals maintain their sense of coherence (i.e., sense 
that life is comprehensible, manageable, and meaningful) and ability to function in 
the face of changes in themselves and their relationships with their environment." 
[150] A similar definition of health that one might adopt is “the ability to adapt and 
self-manage in the face of social, physical, and emotional challenges." [151] 

Yoga and meditation can improve one's ability to generate thoughts that are 
capable of leading to positive emotions. Although cognitive methods by themselves 
can contribute to stress management [152], individuals can achieve better results 
when they integrate emotions into their methods. Recent studies indicate, for 
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instance, that individuals adapt their behaviors more easily when they integrate 
emotions into behavioral change techniques [153]. One reason for this is that posi- 
tive feelings towards a new situation can facilitate physiological and psychological 
acceptance of a change and of the new situation that results. Overall, it has been 
shown that when emotions are a focal point of behavioral change practices, indi- 
viduals are more likely to successfully change their behavior and better able to 
improve their overall QoL. When practiced regularly, yoga and meditation can help 
induce unconscious shifts towards stress-reducing behavior and facilitate synchro- 
nization (coherence states) of a body’s heart rate and brainwaves. 

Yoga and meditation can train one to develop an ability to change perspectives 
by being aware of one’s personal beliefs, which allows individuals to make choices 
regarding how they progress towards change. When individuals are able to change 
perspectives and show competence in empathy, they are ultimately more likely to 
change their behavior. For example, it has been demonstrated that an individual who 
wants to start doing more exercise to improve their QoL, better understands their 
negative emotions during moments when they are not exercising. 

Yoga and meditation can empower individuals to change their perspective 
towards situations that are considered stressful. Some studies have demonstrated 
that one’s attitude towards stress plays a crucial role in the effort to reduce stress. 
Those who consider stress as something positive are more likely to manage it suc- 
cessfully than those who view it negatively. 

Self-observation with meditation and yoga helps to raise awareness and activate 
unconscious processes; raised awareness is a result of logging emotional states [154]. 
Meanwhile, developing an emotion-focused understanding of stress allows one to 
better understand what is happening when one is dealing with undesired behaviors. 
Yoga and meditation increase awareness and understanding at critical behavioral 
moments. Meditation in particular also helps individuals to enter a state in which they 
are less focused on thoughts so that they can approach their emotions as an observer. 

Individuals who wish to empower themselves may utilize technology for the 
self-assessment of their states and behaviors. As discussed in the previous section, 
miniaturized devices such as smartphones and wearables can be used to gather rel- 
evant quantitative data. The result of using these tools is what is called, in the study 
of technology, the quantified self [155], a term that equally describes the practitio- 
ner of yoga and meditation. There exists a myriad of personal digital devices [156] 
that can automatically gather information regarding physiological processes and 
hard-to-observe behaviors [156]. The use of these devices and the data they provide 
may motivate individuals to better understand the way certain thoughts, beliefs or 
feelings lead to specific behaviors. Researchers have found that individuals find it 
easier to internalize such knowledge when using technology and, consequently, are 
able to realize behavioral changes almost instantly [154]. In general, the idea that 
technology can enable the assessment of individual behaviors and health and pro- 
duce improvements in life quality has been expressed before [157]. The next step in 
the process is to leverage these technologies to potentially focus on the assessment 
and improvement of the individual’s internal states—particularly their thoughts, 
beliefs and emotions. 
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Concluding Remarks 


In this chapter, we have examined the role of personal beliefs, together with thoughts 
and emotions, in the development of behavioral habits. Considering stress as a reac- 
tion that is influenced by behavior, we discussed yoga and meditation practices as 
possible interventions to alter personal beliefs and influence behaviors and thereby 
reduce stress, leading ultimately to an improvement in QoL. In addition, we sug- 
gested the use of technological devices and other tools for quantifying thoughts, 
beliefs, emotions and behaviors as a means of further empowering behavioral change. 

Many benefits of yoga and meditation have been uncovered in recent research, 
including their effect on stress reduction. It has been specifically shown that manag- 
ing stress and adapting one’s behavior to live a more meaningful and fulfilled, 
healthier and happier life, depends very much on each individual’s thoughts, their 
personal beliefs and related emotions. Yoga and meditation, as techniques in which 
individuals seek to understand their perceptions and reactions towards stress and the 
outer world, are promising, learnable skills that can lead towards self-empowered 
behavioral change. Once the body has learned these skills, it reacts (i.e., thinks, 
believes, feels and behaves) unconsciously and without effort, even when the indi- 
vidual is not consciously paying attention. Studies have shown that, in such cases, 
behavioral change can then occur unconsciously. 

Scientific research is now beginning to map the bioenergetic communication sys- 
tems between the "inside" and “outside” of the body (including physiological func- 
tions, cognitive processes, emotions and behavior) that influence one's health, 
happiness and QoL in the long term [158]. More research is expected to be carried 
out in the future concerning this important, interesting and impactful topic. 
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Chapter 13 A 
Technology-Enabled Assessment crente; 
and Improvement of Inclusive Learning 
and Quality of Life in Higher Education 


Maria Toledo-Rodriguez and Thomas Boillat 


Introduction 


Universities have, for centuries, been the sources of wisdom and knowledge. In 
order to acquire this knowledge students used to travel to the universities where 
lecturers, masters in their field, would impart their teachings. Often the transfer of 
knowledge consisted in masters giving hour’s long monologues while the students 
passively listened and took notes. In order to be successful, this system requires 
from the student to: (a) attend to all learning sessions, (b) be able to record all the 
knowledge provided while listening to the lecturer and (c) understand and put in 
context (within their applications and implications) all information passively 
received during that unique offering of knowledge. This system puts in clear dis- 
advantage many students who have learning differences, suffer from poor physical 
or mental health or have additional family/work commitments. Quality of higher 
education is recognized by researchers and governments to impact the student’s 
quality of life in the long term [1, 2]. The term “inclusive learning” is currently 
being widely used by academics and governments as a tool to improve the learning 
in those populations and therefore potentially improve the long-term quality of life 
of these students and dependants. However, inclusive learning as described by 
May and Bridger”... necessitates a shift away from supporting specific student 
groups through a discrete set of policies or time-bound interventions, towards 


M. Toledo-Rodriguez () 
School of Life Sciences, University of Nottingham, Nottingham, UK 
e-mail: maria.toledo @ nottingham.ac.uk 


T. Boillat 

Design Lab, Mohammed Bin Rashid University of Medicine and Health Sciences, 
Dubai, UAE 

e-mail: Thomas.boillat ? mbru.ac.ae 


© The Author(s) 2022 319 
K. Wac, S. Wulfovich (eds.), Quantifying Quality of Life, Health Informatics, 
https://doi.org/10.1007/978-3-030-94212-0_13 


320 M. Toledo-Rodriguez and T. Boillat 


equity considerations being embedded within all functions of the institution and 
treated as an ongoing process of quality enhancement” [3]. In the following sub- 
sections we define the current technological solutions to enable holistic inclusive 
learning. 

This book chapter presents results of a study employing mixed methods and will 
focus on the role of different digital technology platforms in supporting assessment 
and improvement of inclusive learning in higher education. However, it does not 
systematically focus in detail on software specially developed to help students with 
learning difficulties. Rather, it presents and analyses recent advances in learning 
technologies that bring opportunities for inclusive learning and enable individuals 
to tailor the learning experience to their needs and abilities regardless of their degree 
of disability. Most of these platforms analyzed in here are relatively recent and 
while they charge per use, the majority also have a free version, which enables any 
educator to make use of them (with some limitations), as long as students and teach- 
ers have internet connection and a mobile device. In this chapter, we present results 
of our analysis and discuss design implications for technologies, which, if imple- 
mented effectively, will influence the assessment and improvement of learning and 
attainment of educational goals, as well as quality of life in the long term of any 
student. 


Current Assessment Methods for Cognition/Executive 
Function, Attention, Memory and Learning 
in Healthy Populations 


Before focusing on digital platforms that enable inclusive learning, we should take 
a quick look at the current assessment methods of cognition, executive function, 
including attention and memory and learning in healthy populations. The list of cur- 
rently used tests used is very large and describing it into detail would take us away 
from the focus of this chapter. Table 13.1 describes a representative number of tests 
for executive function, attention, intelligence and different types of memory and 
academic resilience. We focused on tests that measure performance (PerfRO) [32]. 
Regarding learning, the best methods to test healthy populations are current aca- 
demic assessments (e.g. written exams, coursework, oral presentations, group pre- 
sentations, etc.). 
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Visual Tools 


| Tools Consist in S 


Executive function 


Approach/avoidance task (ATT) | Measures behavior when faced with a conflicting | [33] 
goal or event (which has both positive and 
| negative characteristics or effects). 


Working memory 
Automated Working Memory Fully automated online assessment of working [34] 
Assessment (AWMA) | memory. 


Functional capacity/Executive function 


Virtual Reality Functional Capacity | Immersive virtual reality interactive gaming that | [35] 
Assessment Tool (VRFCAT) measures functional abilities using realistic 
| simulations of daily environments. 


In this paper we focus on means of technology-enabled assessment and improve- 
ment of inclusive learning and quality of life in higher education. 


Common Challenges to Inclusive Learning 
in Higher Education 


Student’s ability to learn and fulfil their educational goals is affected by different, 
physical, mental (as defined by DSM-5 [36]) and situational challenges. It is esti- 
mated that on the UK, 3.5% of full time university students had a mental health 
condition in the 2017-2018 academic year. We will now discuss how each one of 
them impacts learning. 


Physical Health and Illnesses 


During their university studies a student might have periods of ill physical health or 
hospitalization. While in most cases the physical illness has a short duration (e.g., 
flu), some students might have long periods of illness (e.g., broken arm) or might 
live with chronic conditions (e.g., Chronic Fatigue Syndrome). In those cases, the 
student might not be able to attend university for periods of time. As a result, they 
won't be able to take part in lectures, practical sessions or exams. They can also not 
engage with in-class activities, such as peer-assessment or debates. Additionally, 
students will struggle to submit paper-based coursework (although they may be able 
to submit coursework online). Moreover, if they are in pain (e.g., broken leg) the 
pain and/or pain medication might interfere with their concentration and sleep pat- 
terns. Finally, the injury or illness might prevent students from typing or taking notes. 
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Mental Health and Illnesses 


While it is normal to feel sad or anxious at times, clinical depression and anxiety 
disorders can be crippling conditions, which can render the student unable to 
engage with their studies. Recent data shows that the mental health of students in 
higher education is deteriorating [37]. For example, a recent study from the 
Institute for Public Policy Research, reports that the number of first year students 
reporting a mental health problem in 2016 was 3 times higher than 10 years earlier 
in the UK [38]. As a result, there has been an increase in demand of Counselling 
and Mental Health services putting huge pressure on the universities’ and local 
welfare resources. In order to deal with this challenge, universities, governments 
and charities are trying to increase their mental health provision and there is a ris- 
ing number of preventive campaigns. Additionally, there is an increase interest in 
initiatives, which aim to embed mental wellbeing within the curriculum [39]. 
These initiatives can be very varied, encompassing, from university-wide curricu- 
lum infusion, such as the Engelhard project at Georgetown University, USA [40] 
to focus on processes that might cause undue stress or disadvantage to some stu- 
dents (e.g. the use of lecture capture to revise lectures they were not well enough 
to attend or the use of anonymous online interactive tools to eliminate social 
anxiety). 

The most common mental health disorders experienced by undergraduate stu- 
dents are depressive disorders (e.g. major depressive disorder, bipolar disorder or 
seasonal affective disorder), and anxiety disorders (e.g. general anxiety disorder or 
social anxiety disorder) [41]. A small percentage of university-age individuals expe- 
rience other types of mental health disorders such as schizophrenia [41]. 

Depression is a common mood disorder that is characterized by constant sad- 
ness, lack of hope, motivation and energy, disturbed sleep patterns, and difficulties 
to make decisions [36]. As a result students feel that doing every day tasks is a 
constant struggle; they feel worthless and believe that their actions are useless and 
so their engagement with their studies suffers (they may stop attending lectures or 
might not be able to keep up with coursework or revision). At times they can become 
angry because they feel judged by others and themselves and believe they are falling 
short of expectations all the time. Their view of the world is pessimistic and they 
believe their situation will never improve. In extreme cases the student might plan 
and/or attempt to take their own life, although the percentage is lower compared to 
the general population of similar ages [42]. 

Bipolar disorder is characterized by periods when the student is depressed (see 
above for symptoms) alternated with “manic” phases, when the student is overex- 
cited and overactive, but often doing tasks that are not productive nor make sense 
(e.g., writing for hours an essay about how to tickle ants when the assignment was 
on the habitat of insects that live in colonies) [36]. 

Seasonal Affective Disorder is very similar to depression, but usually only hap- 
pens during a time of the year (e.g. in the European context, commonly during 
winter, when the days are short and often there is lack of sun) [36, 43]. Often stu- 
dents find particularly difficult to wake up on time in the morning, and have low 
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energy levels during the day. As a result students might stop attending lectures and/ 
or struggle to keep up with studies. 

Generalised anxiety disorder is characterised by constant worry which leads to 
difficulties in concentration and/or sleep. In extreme cases students might develop 
panic attacks (for example in an exam) [36]. 

Students suffering from social anxiety worry disproportionately about social 
activities, such as going to class or speaking in public [36]. They feel judged at all 
times by others and fear that they will do something in public that is inappropriate. 
They are scared of criticism and can become paralised when others are watching 
what they are doing (such as giving a public presentation). In extreme cases students 
might develop social phobia, where they avoid interacting with others or speaking 
in public and/or experience panic attacks. 

Schizophrenia is a severe mental disorder characterised by relapsing episodes of 
hallucinations (hearing or seeing things that do not exist outside their mind), disor- 
dered thinking, social withdrawal and cognitive impairment [36]. Students with 
schizophrenia may react in “odd ways”, sometimes believing that staff and/or other 
students are being bullies. 


Learning Differences, Neurodiversity 


Students’ intellectual ability can vary widely resulting in some cases in being diag- 
nosed with a learning difference/disability (neurodiversity). Specific learning disor- 
der is defined by DSM 5 as “a neurodevelopmental disorder with a biological origin 
that is the basis for abnormalities at a cognitive level” [36]. One essential feature of 
specific learning disorder is persistent difficulties learning keystone academic skills. 
In the 2015-2016 academic year, the Higher Education Statistics Agency reported 
4.95% of all students in the UK enrolled in all higher education had a Specific 
Learning Difficulty [44]. These disorders are often diagnosed in childhood and are 
managed with the help of school teachers/advisers and/or specialised software/tech- 
niques. However, when students reach university, they might struggle with the tran- 
sition into a less structured and/or supportive environment. University students with 
learning differences can use specialized assertive and adaptive technologies, which 
compensate for the decreased ability (such as Texthelp! or ClaroView? and 
ScreenRuler,’ which can help them reach their full academic potential [45]. There is 
a spectrum of learning differences as discussed below. 

Students in the Autism spectrum or Aspergers find it hard to figure out what 
others are thinking or feeling based on their facial expressions and/or words [36]. 
Because of this they find socialising difficult. Noise or crowded spaces might be 
overwhelming for somebody with autism and thus sometimes they need to retire to 
a quiet space in order to calm down (they may also sit close to the entrance in class 


! https://www.texthelp.com/ 
?https://www.clarosoftware.com/ 
5 https://www.clarosoftware.com/ 
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so that they can leave easily or might avoid crowded/noisy classrooms). They can 
seem awkward or feel uncomfortable in social situations, not making eye contact 
(resulting in appearing rude or patronising). Often they may take literally what is 
told to them (so they might struggle to understand jokes or sarcasm). Additionally, 
changes in their routine can cause them anxiety. Because of this they are very keen 
on having clear plans for everything that has to happen. 

Students with dyslexia often mix up letters that have some similarity (like “b” 
and *d") or words that have very similar spelling (e.g. “from” and “form’’). As a 
result they often make spelling mistakes and are very slow readers/writers. They 
avoid reading and/or writing whenever they can (e.g. they will prefer a presentation 
than a written exam or coursework). They also struggle to write coherent notes dur- 
ing lectures (as they struggle to write while trying to understand what the lecturer is 
explaining and reading from the slides). However students with dyslexia can easily 
understand information when it is spoken to them and can explain themselves orally 
much clearer than in writing. Sometimes they struggle to follow directions (sequence 
of things that have to be done in a certain order) and thus find it difficult to organise 
and plan, which compound with their writing challenge results in students strug- 
gling to meet coursework deadlines [46]. 

Although Attention Deficit Hyperactivity Disorder (ADHD) is not considered 
a learning disability, it has at times a profound impact on the ability of the student 
to learn [36]. ADHD is characterised by difficulties in focusing (and lack of atten- 
tion to details) and hyperactivity (which might make attending to long lecturers, of 
finishing coursework a challenge). At times, a student with ADHD might have dif- 
ficulties coping with stress or remembering. 

We should note that learning disabilities often do not sit in binary silos where the 
student either has a learning disability or not (with students often displaying differ- 
ent grades of a learning disability) [36]. Therefore, scholars are starting to strongly 
argue against the concept of “the average or typical student”. They reason that the 
students traditionally identified as having learning differences belong to the extremes 
of a continuum [47] and that we could consider that all students abilities are scat- 
tered across a spectrum of learning differences [48]. 


Students Juggling Higher Education with Work or Caring/ 
Family Commitments 


The majority of university students are relatively young (between late teens to mid 
20s), do not have caring responsibilities (such as caring for young children or elder 
relatives) and do not need to work in order to fund their studies and daily life [49]. 
However, due to recent economic downturns and drastic increase in university fees, 
many students in higher education have to work while they study, in order to pro- 
vide for themselves and/or finance their studies [50]. Moreover, due to an increased 
diversity in student population and the development of graduate entry courses, there 
is a slight rise in the number of university students with family and/or caring com- 
mitments, particularly within some vocational disciplines such as nursing [51]. 
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These commitments might at times clash with their studies, preventing them from 
attending university (as they have to work or take care of somebody). Thus, at times, 
they may not be able attend lectures, practicals or exams. They might also struggle 
to submit paper-based assessments and can not benefit from taking part in in-class 
activities. 


Method 


In view of answering our research questions 2 and 4, we relied upon a mixed meth- 
ods research approach based on literature reviews, surveys and interviews as shown 
in Fig. 13.1. 

To get an accurate understanding of the problem definition, the research started 
with (a) reviewing the most common learning challenges using the Diagnostic and 
Statistical Manual of Mental Disorders (DS8M—5) as primary source and selected 
literature as secondary source (presented in the previous section). We then built 
upon a (b) recent literature review [52] to gain a deeper understanding on the role of 
information technology (IT) in education as well as a description of digital learning 
platforms and the evolution of learning management systems (LMS). (c) We then 
surveyed eight faculty members (see table below) to gather current uses of digital 
learning platforms (Table 13.2). We asked them the three following open-ended 
questions: (qı) “What digital platforms or tools do you use for your lectures?" (q2) 
“What makes these platforms or tools attractive?" and (q3) “What other platforms 
would you recommend and why?" (d) By means of interviews, we then asked the 
same questions to one course director and three learning designers to gather their 
experiences of state-of-the-art tools for inclusive learning. The interviews allowed 
us to derive four digital learning tools categories, categorized independently by the 
co-authors of this paper. (e) In order to analyze the commonalities and differences 
across these four categories, we used the Business-Application-Information- 
Technology (BAIT) model. We then summarized our findings by refining the seven 
principles of the Universal Instructional Design [53, 54] framework for the context 
of inclusive learning. 


Research °) » P 2 , 
activities 
Outcomes  Listofthe most common Role of IT in education, List of learning tools Four learning tool BAIT analysis 
challenges in inclusive definition of digital learning and platforms used for categories, list of trends 
learning platforms, evolution of LMS teaching 


Refinement of the7 UID principles for the context of 
inclusive learning 


Fig. 13.1 Research methods 
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Table 13.2 Profiles of survey participants in step (c) and (d) 


Academic position Country Current class size Educational area 
Associate professor, 2 years of TE? | United Arab 40-65 BSc students | Medicine 
Emirates 
Assistant professor, 4 years of TE | United Arab 25-65 BSc students | Medicine 
Emirates 
Professor and chair, 20 years of TE | Switzerland 30-200 BSc and Business 
MSc students informatics 
Assistant professor, 5 years of TE | Switzerland 30-70 BSc and MSc | Business 
students informatics 
Professor, 12 years of TE Switzerland 50-300 BSc and Economics 
MSc students 
Teaching Fellow, 2 years of TE United Kingdom | 100-300 BSc and Life Science 
MSc students 
Associate Professor, 6 years of TE | United Kingdom | 20-200 BSc and Life Science 
MSc students 
Assistant Professor, 14 years of TE | United Kingdom | 15-200 BSc and Life Science 
MSc students 
Course director, 7 years of TE United Kingdom | 45 Foundation year | Life Science 
(+15 years in secondary school students 
teaching) 
Senior learning designer, 8 years | United Arab 300 BSc and MSc Health Sciences 
of TE Emirates students 
Senior learning designer, 6 years | United Arab 250 BSc and MSc Health Sciences 
of TE Emirates students 
Learning designer, 4 years of TE | United Arab 250 BSc and MSc Health Sciences 
Emirates students 


a TE Teaching Experience 


Digital Learning and Teaching Platforms (DLTP) Description 


and Analysis 


Along with the almost universal access to personal computers and smartphones, 
Internet and web applications have disrupted education [52]. Not only have they 
changed the way knowledge is transferred and learnt, but also the dynamics between 
lecturers and students as well as among the students. From our interviews with 
learning designers (Table 13.2), four categories of digital learning and teaching plat- 
forms have emerged, as potential enablers for the assessment and improvement of 
inclusive learning: Learning Management Systems (LMS), Social and collaborative 
tools, In-class interactions and Out-of-class interactions. 


Learning Management Systems 


LMS are software applications designed with the specific intent of assisting instruc- 
tors in meeting their pedagogical goals of delivering learning content to students 
[55]. Available as web application by means of an Internet browser (Figs. 13.2 and 
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Fig. 13.2. Canvas LMS—Assignments View of a Class (Under the license: https://creativecom- 
mons.org/licenses/by-sa/3.0/deed.en) 


13.3), LMS have disrupted the way students access (digital) lectures, communicate 
with their lecturers, classmates and other learning communities, access course 
materials, take online quizzes and submit their assignments [56]. From the lectur- 
er’s point of view, LMS allow to organize classes, publish course materials, create 
and grade assignments among many additional features. Our interviews revealed 
two types of software applications: Open-source with Moodle* being on top of the 
list and Graasp? in the role of outsider [57]. On the proprietary side, Canvas was the 
most cited. In our case, LMS were used to structure and describe lectures, upload 
and share documents, lecture recordings and tutorials as well as post and submit 
assignments. Uploaded files are presentation slides, PDF and text documents. Our 
survey revealed that LMS are mostly accessed via desktop or laptop computers. 


Social and Collaborative Tools 


Social and collaborative tools are defined as (web-based) applications that allow 
users to create and to share online documents, spreadsheets, presentations, and 
forms [58]. They are mostly used by students to create and share content such as 
assignments. Similar to LMS, open-source and proprietary tools are being used. The 


^https://moodle.com 
Shttps://graasp.eu 
°https://www.instructure.com/canvas/ 
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Fig. 13.3 Moodle—Class View 


two most used are respectively Google Docs’ and Microsoft Office 365.8 They allow 
students to create different types of documents from presentations to spreadsheets 
and letters and work simultaneously on the same documents (Fig. 13.4). As part of 
Microsoft Office 365, Teams offers new opportunities for lecturers to engage with 
students, and also for creating groups amongst students. It provides instant messag- 
ing capabilities, both via text and video, enabling lecturers and students to commu- 
nicate regardless of their location. In addition, it offers live captions to help students 
with hearing impairments. Newcomers such as Padlet? are increasingly used in class 
for students to publish their work and collect feedback (Fig. 13.5). These tools are 
most of the time accessed from a desktop or laptop computer while a mobile inter- 
face is also available but difficult to use. 


In-Class Engagement 


As stated in Holzer et al. 2013 [57], interactivity in person classrooms is considered 
an important success factor in learning but it remains very challenging to promote. 
Simple technologies such as the Clickers (that combine hardware with software) 
gained popularity in classrooms for their ability to gather anonymous answers to 
questions asked in class [59]. It gives the lecturer a chance to immediately assess the 


Thttps://docs.google.com 
Shttps://www.office.com/ 
?https://padlet.com 


13 Technology-Enabled Assessment and Improvement of Inclusive Learning and... 333 


Collaborative editing co (EES 
Fle Edt View inset Format ‘Tools Table Help All changes saved in Dive "tem a 
Wo 0 TP Normal tent Aral " 8S7 V A-B- oP EE WE HM IB- i-i B8 X Li 


ki, I am creating a short how-to edit collaboratively. 


1. Goto this fink: 
https //en wikipedia org/wiindex php 7title=Speciat UserLogin&returnto-Main *Page&ty 
pe-signup 

2. Give a unique usemame, type a password, retype, type the jumbled letters you in the 
captcha in box there and your email address (it's optional, provide it only if you want 
others could contact you via mails, and wikipedia won't spam 


Fig. 13.4 Padlet—Main View (shared amongst groups or students) 


MM. 
DIT 


[z] 
Advanced/IB Film 2019-2020 


Unit1-FilmasanArt = innis i - H Unit 5- Soviet Montage : 
M : t 
: 


Serenity: An Exploration of 
Genre 


This is a fantastic look into how a j s ~ LUZES% 
quumeten nen voe e L 5 5 CIDADE 


Western/Sci-fi works. =e" ` 


New Film Of The Week - Battleship 
Potemkin 


o. > 


19 


Fig. 13.5 Google Doc—Collaborative work (Under the license: https://creativecommons.org/ 
licenses/by-sa/3.0/deed.en) 


students’ knowledge and quantitatively measure the progress of the class (Fig. 13.6). 
The in-class engagement tools are seen as the next generation of interactions by 
allowing lecturers and students to send anonymous text messages in addition to 
creating polls (Fig. 13.7). They are mostly used by lecturers to gather feedback from 
students and to encourage them to engage in lectures/workshops/seminars. A few 
tools were revealed by the survey: Speakup,'° which was developed by two Swiss 


'http://speakup.info 
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Fig. 13.8 Ms. Teams—Discussion panel 


universities and available free of charge; Socrative!! and Slido? that offer similar 
functionalities than Speakup; Mentimeter? that also allows for the creation of pools 
but requires a subscription for intensive usage; and Kahoot" that leverages the con- 
cept of gamification. All these tools are available as web and mobile applications 
and are most commonly used from a smartphone and laptops. 


Out-of-Class Interactions 


Out-of-class communication channels widely depend upon the university, the fac- 
ulty/school (discipline) and the culture. Most of the course communication and 
feedback relies on emails or on LMS (e.g., class forums). Tools such as MS Teams, ^ 
Speakup or Slido are also used after the class whereby lecturers or students give 
feedback (Fig. 13.8). Recently Whatsapp!^ and other text message applications have 
become very popular amongst students to exchange tips and documents. These lat- 
ter tools are easier to access and to use, thanks to their integration in smartphones. 


! https://www.socrative.com 

P https://www.sli.do 

P https://www.mentimeter.com 

14 https://kahoot.com 

P https://www.microsoft.com/en-ww/microsoft-365/microsoft-teams/group-chat-software 
l^https://www.whatsapp.com 
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Social and 
collaborative In-class Out-of-class 
LMS tools engagement interactions 
Teaching Manage classes, | Create and share | Assess knowledge | Share or ask 
processes documents and | content and understanding | course information 
assignments 
Application Moodle, Canvas, | Office 365, Speakup, LMS, whatsapp, 
Graasp Google Docs, mentimeter, emails 
Padlet Kahoot, Socrative, 
Slido 
Information Lectures, Assignments Course-related Course-related 
readings, (content) questions and catch up 
tutorials, answers information 
assignment 
(description) 
Files PDF, PPT, DOC, PPT, PDF | N/A (text messages) 
videos, images photos, videos 
Technologies Web applications | Web applications | Web applications, | Web applications, 
mobile mobile 
applications applications 
Mediums Desktop and Desktop and Desktop and laptop | Smartphones 
laptop computers | laptop computers, (desktop and 
computers, smartphones laptop computers) 
tablets 


Within and Cross DLPT Analysis 

Table 13.3 summarizes and analyzes the differences and commonalities of the iden- 
tified DLTP. As analysis framework, we used the Business-Application-Information- 
Technology—BAIT model [60]. The latter is used to logically and functionally 
describe a system and its dependencies. More specifically, the Business layer looks 
at the business rationale and the processes that follow from it; the Application layer 
investigates the organization of capabilities and functions; the Information layer 
looks at the data models; and the Technology layer investigates the implementation 
of logic, standards, bundling and tooling. For a more distinct analysis across the 
DLTP, we adapted the BAIT model as follows: The Business layer became the 
teaching process. We added a File layer to represent the types of file managed by the 
system and a Medium layer to be able to better differentiate the medium used to 
execute the application. 


Assessment and Improvement of Inclusive Learning 
with Digital Learning Platforms (DLTP) 


The analysis above sheds light on the types of application used by a selected sample 
of lecturers. Though it helps identify what applications and functionalities are the 
most popular DLTPs, the following critical question remains: How much do these 
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learning platforms and applications support inclusive learning, and how it can be 
evaluated based on the data originating from these platforms? To answer it, we pro- 
ceeded as follows: First, we extracted the challenges linked to inclusive learning as 
described in the section above. Second, we classified the challenges along three 
impairments—.e., physical that refers to a student's physical limitation to follow a 
course; cognitive that refers to a student's potential troubles to remember, learn new 
things, concentrate, or take decisions and behavioral that refers to inability for a 
student to build or maintain satisfactory interpersonal relationships with his or her 
classmates and lecturers (Table 13.4). Third, we mapped these impairments to func- 
tionalities provided by digital learning platforms and tools and reasoned upon their 
assessment functionalities based on the data originating from the application logs 
(Table 13.5). Finally, we suggested potential implications and functionalities for 
some of the unmet challenges based on our focus groups with learning designers 
(N = 3). 


Physical Impairments 


Physical impairments are divided into two subcategories: (a) The inability to attend 
classes or exams, caused by injuries, care commitments or illness, and (b) some 
limitations that prevent students to hear, type or see. For the former (a), DLTPs 
should allow students to access teaching materials regardless of their location, as 
long as they have an Internet access. The policy with regards to video-recording of 
the lectures depends very much upon the university. In some cases, all classes are 
recorded, while the opposite is also true. Traditional universities are afraid that stu- 
dents would not attend classes if they had online access to such content. Given that 
some students might not return to classes for several months, the ability to view 
lectures and write exams remotely is critical. Software applications such as 
Examsoft!" or Rogo! that provide an Internet-based safe environment are increas- 
ingly used by universities (particularly during the Covid pandemic). On the other 
hand, students suffering from hearing, typing or visual impairments (b) are not 
offered much help from DLTPs. Though collaborative tools such as Office 365 
embed “read aloud" and “dictate” functionalities for Word and Excel, it is often not 
the case for slide-decks (e.g., PowerPoint) that require the lecturers to manually add 
captions. Similar problems can occur with PDFs documents depending on the origi- 
nal format and the conversion process. Moreover, video captions are also very rarely 
added by lecturers and when added the caption software can make mistakes (par- 
ticularly when scientific terms and acronyms are used). However, some applications 
such as Microsoft Teams allow for live captions. Lastly, students who have difficul- 
ties using a keyboard do not receive much support from DLTPs. Although there 


 https://examsoft.com 
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Table 13.5 Map between system requirements and DLTPs 
Social and 
collaborative In-class Out-of-class 
R# | LMS tools engagement interaction Opportunities 
1 | Asynchronous  |Offline Software to ask | Use of online Wearables such 
teaching (e.g. interactions or answer tools to interact | as Apple Watch? 
video recording | (reduce stress | questions that do not or FitBit Sense” 
of lectures, so of in-preson anonymously | require in person | embed 
that student can | interactions) meeting mechanism to 
stop/rewind detect stress using 
when needed different 
and does do not biomarkers. There 
stress them also exists 
because they dedicated mobile 
might not apps, building on 
understand a scientific 
detail) research, to assess 
and reduce stress 
[61]. More 
commercial 
solutions are also 
on the market 
such as 
Headspace* 
2 | Asynchronous Some 
teaching (e.g. scientifically 
video recording tested mobile 
of lectures, so applications allow 
that student does the assessment 
not need to and management 
come in person of anger, notably 
to a room where based on Remote 
they might Exercises for 
become angry Learning Anger 
with others) and Excitation 
(RELAX) [62]. 
3 | Asynchronous Several mobile 


teaching (e.g. 
video recording 
of lectures, 
enable student to 
study the 
material when 
medication has 
helped reduce 
the psychosis) 


apps allow for the 
assessment and 
treatment of 
hallucinations 


[63]. 
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Social and 
collaborative In-class Out-of-class 
R# | LMS tools engagement interaction Opportunities 
4,5 | Asynchronous | Offline Software to ask | Use of online 
teaching (e.g. interactions and answer tools to interact 
video recording | (reduce questions that do not 
of lectures, so disruption online will require in person 
that student does | cause by odd enable student | meeting (e.g. 
not need to behaviour, e.g. | to participate small group 
attend in person, | tourette regardless of meetings using 
reducing syndrome) ticks or other | MS Teams where 
chances to odd behaviour | cameras are off 
interrupt the so student does 
study of other not have to worry 
students because about suppressing 
of their odd ticks or other odd 
behaviour) behaviour) 
6 Software to ask 
or answer 
questions 
anonymously 
7 There exist many 
mobile apps that 
help assess the 
condition and 
suggest digital 
therapies [64]. 
7 Distance Students and 
learning is teachers can 
enabled by interact via 
means of slide | video or text 
sharing and live | messages. 
sessions. 
8,9 | Additional Bespoke apps/ 
modules allow webpages for 
for protected online 
and secured examination (e.g. 
online exams Rogo) 
10 Voice captures Text messaging 


can be 
automatically 
generated (from 
video too), 
while image 
captures require 
manual 
description. 


software/tools 
enable off class 
interaction with 
peers 
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Table 13.5 (continued) 


Social and 

collaborative In-class Out-of-class 
R# | LMS tools engagement interaction Opportunities 
11 Platforms such 

as Office365 

embed “read 

aloud” 

throughout its 

applications. 

Still limited 

with pictures or 

other media. 

12 Specialized 
dictation 
software? 

13 |Class material Mobile apps,° 
(including scientifically 
videos) can be tested can help 
accessed away student focus 
from the 
classroom (in a 
quiet place with 
less distractions) 
and at a time 
when the student 
is less tired 

14 |Class material Mindmap 
(including software to 
videos) can be organize the 
accessed knowledge more 
multiple times to visually! 
enable revising 
multiple times 
content 

Several apps have 
shown to be 
efficient to assess, 
improve 
information 
retention and 
exercise the 
memory [65, 66]. 
15 Software (like Apps can support 


Office365) help 
with spelling 
and grammar 


the assessment 
and provide 
exercises to 
improve the 
condition [67]. 
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Table 13.5 (continued) 


Social and 
collaborative In-class Out-of-class 
R# | LMS tools engagement interaction Opportunities 
16 Platforms such Specialized 
as Office365 dictation 
embed “read software® 
aloud” Content captures 
throughout its as well as speech 
applications recognition apps 
can help with 
providing 
alternative modes. 
Specific apps can 
also be used to 
work on different 
aspect of the 
memory and help 
develop reading 
and writing skills 
[68]. 
17 | Timetables Platforms such Time 
embedded in as Ms. Teams management tools 
LMS can help embed calendar such Rescue 
student create functionality Time^ 
their individual | that can be 
timetable linked to online 
classes 


“https://www.apple.com/ae/watch/ 

Shttps://www. fitbit.com/global/us/products/smarwatches/sense?sku=5 12BKBK 
*https://www.headspace.com/ 

¢https://www.nuance.com/en-gb/dragon.html 
*https://www.frontiersin.org/articles/10.3389/fnbeh.2019.00002/full 
'http://freemind.sourceforge.net/wiki/index.php/Main Page 
Shttps://www.nuance.com/en-gb/dragon.html 
*https://www.lifehack.org/articles/technology/top-15-time-management-apps-and-tools.html 


exist additional software applications such as DragonSystems,'? they would only 
partially fulfill the tasks of a student. As an alternative, in-class engagement apps 
can also be used post-lecture to continue discussions that occurred during lectures. 
To conclude, multimodal access to content is currently missing. 


Cognitive Impairments 


Cognitive impairments refer to students who have difficulties focusing, remember- 
ing and those who are slow readers and mix up letters. It can also be side effects 
from anxiety, stress, depression or medication taken for physical or mental health 


1 https://www.nuance.com/dragon.html 
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disorders such as pain or schizophrenia. In both cases, online access to teaching 
materials and video-recordings in particular, brings many advantages. Students can 
learn at their own pace, moving back and forth until they have acquired the knowl- 
edge. However, LMS and other platforms fail at two levels: (a) in providing tailored 
learning experience, assuming that each student has the same learning capacity, (b) 
in assessing the condition and providing exercises to support the students. Similar 
to guidelines for people suffering from physical impairments (e.g., minimum font- 
size, reduced amount of length, prioritizing images), digital learning platforms have 
the opportunities to implement machine learning driven quizzes that adapt the ques- 
tions to the learners as well as adapt the content presentation (e.g., learning section) 
too. In parallel, students can also use wearables to track their biometrics and thus 
have a more objective assessment of the impairments. 


Behavioral Impairments 


Students with disorders such as depression, anxiety, autism or schizophrenia display 
some behavioral impairments that prevent them from fully engaging with the learn- 
ing. They sometimes have access to support workers that take notes for them or 
extra time during assignments. They also sometimes have access to “safe” environ- 
ments such as dedicated rooms for them to calm down in case of a sensory overload. 
Aside from this, tools such as Speakup offer safe digital space in which students can 
anonymously ask and answer questions without the fear of being individually criti- 
cized or judged. For students whose behavioral impairments prevent them from 
attending lectures or can attend only partially, remote access to teaching materials 
(e.g., video-recordings or slides) significantly increases the students’ chance to suc- 
ceed. In this view LMS plays an important role as central platform for content 
access but also for handing in assessments and receiving potential feedback and 
discussions such as in forums. 


What Do DLTPs Support? What Remains Unsupported? 


In order to analyze the means by which DLTPs address the identified inclusive 
learning conditions summarized in Table 13.4, we extracted the system require- 
ments from the same table and mapped them to functionalities provided by the cur- 
rently used, digital learning platforms and tools. We added a column opportunities 
to demonstrate how technologies beyond our scope of analysis can support inclu- 
sive learning. The results are presented in Table 13.5. The first column of the table, 
R#, refers to the system requirements presented in Table 13.4. We also differentiate 
between technologies that assess or directly address the condition (in dark grey) and 
those that mitigate (in light grey) the condition. For instance, condition R#—diffi- 
culties coping with stress and anxiety: DLTPs offer functionalities to mitigate the 
condition by means of asynchronous teaching that allows students to review the 
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Fig. 13.9 "Accessibility checker" and “screen reader helper" currently embedded within the LSM 
system Moodle 


knowledge as many times as required, and offline and anonymous communication 
that allow students to be protected. On the other hand, wearables and mobile apps 
relying on biometrics and self-reporting allow for a quantitative assessment of the 
level of stress. 

When looking at the table above, it is clear that the coverage of inclusive learn- 
ing's needs by the identified DLTPs is very limited. Most of the platforms rather 
mitigate (dark grey) than assess and improve the different conditions. Apart from 
providing distance learning capabilities required for students who cannot access the 
facilities, the tools fail in supporting behavioral and cognitive impairments. Let us 
take stress problem as an example: though some of the learning designers we inter- 
viewed have setup dedicated pages and modules in LMS to provide links to special- 
ized websites, DLTPs do not support students in assessing and helping with their 
stress overload. However, while a considerable number of techniques, approaches 
and a significant number of mobile applications a at the moment, DLTPs do not 
provide any integration for these applications. For some of the challenges for learn- 
ing, we identified specialized tools/apps that are already on the market. However 
sometimes it might be challenging to embed them within the existing learning tech- 
nologies. Some of the current technologies are starting to address this issue (i.e. 
Moodle “accessibility checker" and “screen reader helper" as shown below 
(Fig. 13.9). 

Additionally, in some cases the challenges faced by students require medical 
intervention (e.g. psychosis or depression). In these cases the teaching apps should 
facilitate learning, but they can not replace medication and/or talking therapies. For 
the majority of the conditions, assessment capabilities are available with external 
applications (light grey). Very often these applications and platforms have been sci- 
entifically evaluated. 


Supporting Inclusive Learning: Implications for Design 
of Future Platforms 


Increasing access to learning has been the focus of a great deal of research. The 
Universal Instructional Design (UID) is certainly the most comprehensive [69]. It 
builds on the concept of Universal Design to identify and eliminate unnecessary 
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barriers to teaching and learning while maintaining academic rigor. UID is there- 
fore described as a process that involves considering the potential needs of all 
learners when designing and delivering instruction. We used the UID as frame- 
work in order to refine the seven UID principles for the purpose of inclusive learn- 
ing. Our inclusive learning principles, presented in Table 13.6, build upon the 
analysis that we presented above as well as the survey and the interviews we 


conducted. 


Table 13.6 Universal Instructional Design applied to inclusive learning 


UID principles 


Application in the context of inclusive learning 


Accessible and fair (equitable) use: 
Instructions should be designed to be useful 
and accessible by people with different 
abilities, respectful of diversity, and with high 
expectations for all students. 


Universities and lecturers should make use of 
video-recordings as well as remote access to 
teaching materials for students who cannot 
physically attend classes. In addition, they 
should use (automatic and accurate) captions 
as much as possible to reach students suffering 
from hearing impairments. 


Flexibility in use, participation and 
presentation: Learning is most effective when 
it is multimodal. Instruction is designed to meet 
the needs of a broad range of learner 
preferences. Students can interact regularly 
with the instructor and their peers. 


Content should be delivered in different ways 
including videos (with audio) and in writing. 
Lecturers should also use digital forums and 
other in-class engagement tools to foster 
exchanges with and among students. 


Straightforward and consistent: Instruction 
is designed in a clear and straightforward 
manner, consistent with user expectations. 
Tools are intuitive. Unnecessary complexity or 
distractions that may detract from the learning 
material or tasks are reduced or eliminated. 


Content (e.g., terminologies, description, 
abbreviations, file names) as well as learning 
objectives should be presented consistently 
across all digital platforms used by the 
lecturers. If applicable, a dictionary of terms 
shall be accessible. Humor or references to 
undocumented content should not be used. 


Information is explicitly presented and 
readily perceived: Course expectations are 
transparent. Instructions are easy to understand. 
Communication is clear. Any barriers to 
receiving or understanding are removed. 
Information may be presented in multiple 
forms. 


One digital platform (e.g., LMS) should be 
used as a central hub to indicate the learners 
what to do and where to find the information. 
Lecturers must choose digital platforms and 
communication channels accessible by all 
students (e.g., cross platforms). 


Supportive learning environment: Instruction 
anticipates that students will make mistakes. 
While instruction recognizes that errors are 
necessary, and if handled properly, present 
powerful learning opportunities. It tries to 
minimize hazards that can lead to irreversible 
errors and failures. Instruction also recognizes 
that systems will fail and things can go 
wrong—thus, a tolerance for error and 
preparation by way of backup are important so 
that learning will not be interrupted. 


Instructions should be designed so to reduce 
the chance of mistakes to its minimum. It starts 
with the use of a widespread vocabulary 
accessible to everyone and continues with a 
session design that ensures that students 
identify potential misunderstandings rapidly. It 
includes the use of self-assessment tools, 
group work, in-class interactions and quizzes 
(which provide a safe environment for 
mistakes to be used as learning opportunities). 
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UID principles 
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| Application in the context of inclusive learning 


Minimize or eliminate unnecessary physical 
effort or requirements: Instruction is designed 
to minimize non-essential physical effort (i.e., 
not related to a learning outcome) in order to 
allow maximum attention to learning. 


Learning space accommodates both students. 
and methods: The learning space is accessible 
and the environment supports multiple 


whenever appropriate. 


Instruction and knowledge must be accessible 
for students regardless of their impairments. 
The lecturers should make use of remote 
access whenever possible. Vocabulary as well 
as instructions must be as clear as possible. 
Students should also be able to hand in 
assignments or exams online, while using a 
system that integrates word processing 


Learning spaces (e.g., method, content, 
discussion, assignment) should be physically 
and mentally accessible by all students. It 


should combine spaces for physical and virtual 

classes, communications and instructions. The 

learning space should adapt according to the 
individual student needs and impairments. 


instruction strategies. 


Discussion 


There is no doubt that information technology is transforming education. When 
used for assessment and improvement of inclusive learning, it offers a wide range of 
opportunities for students suffering from behavioral, physical and cognitive impair- 
ments. Beyond the application’s functionalities and goals, from our data and own 
observations, we identified five key elements for a successful implementation and 
uses of digital platforms and tools that assess and improve the individual’s inclusive 
learning. The discussion part ends with some limitations and a note to the signifi- 
cant positive impact that information technology to enable teaching and learning 
during the Covid19 pandemic. 


Assessment vs. mitigation of the conditions Apart from physical impairments 
that are not necessary expected to be assessed via DLTPs, behavioral and cognitive 
impairments are not assessed by DLTPs as shown in Table 13.4. They rather provide 
limited functionalities to mitigate the conditions such as “read aloud" for students 
suffering from visual impairments. Though logs from DLTPs can be used in order 
to assess the number of interactions between students and teaching material for 
instance, this functionality has its limits. It will not inform the lecturer whether the 
student has read or understood the lecture, but only that he had opened it. However, 
a huge number of opportunities come from mobile applications as well as wearables 
that rely on either biometrics or self-reporting to provide objective assessment of 
the conditions. This is notably the case for stress and anxiety, attention loss and 
memory retention's problems. 


Usability Research has looked into the usability of digital learning platforms and 
learning management systems in particular. As stated by Harrati et al. (2016), 
“Positive user experience is of prime importance for educational learning systems 
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playing vital role for the acceptance, satisfaction and efficiency of academic institu- 
tions" [70]. Usability of digital learning platforms has drawn the attention of many 
researchers after universities and schools complained that the tools do not bring the 
promised outcomes and that they are not used appropriately. For example, it is not 
uncommon for universities to install plugins for improving user interface's ease of 
use. For a successful implementation of digital learning platforms, it is then critical 
for the solution provider to focus on user-centeredness (e.g., lecturer, student). 
Though most LMS embed functionalities to interact in classes and create pools to 
engage with students, all lecturers that we interviewed use additional applications 
for their high usability and accessibility. It is even more important in the case of 
inclusive learning due to students' potential impairments. 


Integration in the lecture design It is not sufficient to equip lecturers and students 
with digital learning platforms and tools. The latter must be fully integrated in the 
course and curriculum design. Video-recordings as well as in-class engagement 
activities must be part of a wider program aligned with learning objectives. 


Integration in the enterprise architecture In-class engagement apps as well as 
collaborative tools are rarely integrated with learning management systems. Beyond 
technical challenges, they require taking additional steps (e.g., different logins, user 
interfaces, terminologies) that become confusing for students with cognitive impair- 
ments. Universities therefore have to ensure that the same terminologies and 
descriptions are used. As mentioned above, one system (e.g., LMS) must play the 
role of central hub from which students can access every other application. 


Platform monitoring Once in place, the adoption and usability of the platforms 
have to be closely monitored, also serving for an assessment of the inclusive learn- 
ing path, i.e., the learning path of the individual. For each impairment category, 
champions amongst students have to be identified and their uses (e.g., logs) ana- 
lyzed. Together with the lecturers, learning designers and software providers, uni- 
versities have to collaborate to ensure that the functionalities meet the student needs. 
Although it might sound trivial, it is much often not the case as highlighted by the 
learning designers we interviewed. 


Assessment and impact on quality of life In their current implementations, digi- 
tal learning platforms are designed to assist rather than assess students’ quality of 
life. Following the Quality of Life Technologies' definition [71], DLPT are limited 
in two areas. First, in quantitatively assessing and informing the students with 
regards to their improvement in any of the three impairments: physical, behavioral 
and cognitive. To date, only the total (sum up) mark for all assessments in a subject/ 
course/module can be used to assess any type of progression. Though time spent on 
lectures, number of submission's attempts, number of typos could potentially be 
used to evaluate a student's progress, this information is not available to students 
nor educators. Second, in integrating data from sensors, activity trackers and mobile 
devices, students (and potentially parents) and teachers have a chance to either 
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detect or follow up on some conditions and quality of life in particular using quan- 
titative data [72]. In addition to questionnaires that are given to students at the 
beginning of the semester to assess e.g., stress and anxiety, the integration of con- 
tinuous data from sensors in the LMS could help students, parents and schools to 
quantitatively analyze students’ progress and more importantly find correlation 
between changes in quality of life and impact of academic results. 


Digital therapeutics Some behavioral conditions such as stress, anxiety and 
depression, digital therapeutics are seen as a new type of (mobile) applications that 
are either used to complement medication or to replace it. Described as “evidence- 
based therapeutic interventions driven by high-quality software programs to pre- 
vent, manage, or treat a medical disorder or disease™ , some of these applications 
have already been approved by the U.S. Food and Drug Administration. Building on 
Cognitive Behavioral Therapy, these applications engage with their users on a daily 
basis while using objective data to assess and customize digital treatments [73]. 
These applications could be seen as an alternative or a complement to counselling 
sessions. 


Overcoming learning ability? Limitations Technology, and in particular the 
apps/programs discussed this chapter, is not the only solution to facilitate inclusive 
learning. Often students benefit from a combination of multiple high-tech and low- 
tech approaches, tailored to their unique needs. For example, there is software spe- 
cially developed to help students with learning difficulties (such as Optical Character 
Recognition software to help students with dyslexia or talking calculators to help 
students with dyscalculia), which has been widely reviewed on [45, 74]. Another 
example is granting extension on coursework deadlines to enable students with dys- 
lexia the extra time they may need for proof reading or to provide a student with a 
broken arm time for their arm to heal, so that they can write the coursework. 
Additionally, students with mental health disorders (poor mental health) often 
require medical treatment aimed to heal their condition, such as medication and/or 
counselling, which has little to do with the platform capabilities. Another word of 
caution included a comment that “There are many other apps, but they tend to 
behave similarly to the ones we analyzed” (comments from a learning designer), 
indicating that the platforms need proof of authenticity. 


Impact of information technology on teaching and learning during the Covid19 
pandemic Although most of the content of this book chapter was written before 
Covid19, we could not conclude this chapter without referring to the role that infor- 
mation technologies have had in ensuring a seamless transition into online-only 
teaching during the pandemic. Within weeks universities had to migrate all their 
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learning and teaching activities to online-only provision. From discussions with col- 
leagues across the world we have learn that level of previous familiarity with LMS 
and applications (and “willingness to change" of members of staff) impacted the 
speed and success of the migration. However, regardless of previous preparation, all 
universities had to transfer as much teaching as they could online, with MS Teams 
suddenly being used by lecturers and students across the globe, often with just hours 
or minutes to learn how to use it. An extreme case was seen at the School of 
Veterinary Medicine and Science, Nottingham University (UK), which achieve a 
full-online start of the academic year for their April 2020 intake, with lecturers hav- 
ing only a few days' notice to migrate online their teaching delivery. At the time of 
writing of this article, we could see that across the world academics have adapted 
their courses to distance and Covid-secure teaching, using a mixture of online deliv- 
ery/assessment for any subject that can be taught remotely and in person Covid- 
secure teaching for activities that must happen in person, e.g. nursing practical 
teaching [75]. 


Conclusion 


Universities have a long history of trying to offer education to students who have 
different needs [76, 77]. However, regardless the technology used or the teaching 
method, it is interesting to observe that students sometimes complain; for them a 
"proper" lecture is sitting on a chair for a couple of hours while listening to the 
lecturer's monologue that tells them what they need to know in order to pass the 
exam. Though digital learning and teaching platforms do support assessment and 
improvement of inclusive learning and teaching, which may influence the wellbeing 
and quality of life of the students, for a sustainable impact, changes in mindsets 
have to take place for both, lecturers and students. These platforms became an unex- 
pected blessing and saving grace when the world faced the Covid19 pandemic. We 
are yet to know the real impact of these new practices on the long term learning 
outcomes and life quality of the individuals influenced by these radical and unex- 
pected changes. 
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Introduction 


Maintaining personal relationships represents an integral part of one’s self-care and 
thus contributes significantly to one’s overall health. According to self-determination 
theory, relatedness is one of the three basic psychological needs humans possess [1]. 
Maintaining constructive and healthy personal relationships is a way of satisfying 
this core psychological need. Continuously neglecting one’s personal relationships 
and relatedness with others can potentially cause various psychopathological symp- 
toms and prove detrimental to overall health. This chapter will focus on personal 
relationships in the context of quality of life (QoL) and new digital technologies, 
highlighting the potential benefits of implementing these technologies in research 
on these issues as well as limitations that may slow their widespread adoption. 

The World Health Organization (WHO) defines personal relationship quality as 
“the extent to which people feel the companionship, love, and support they desire 
from the intimate relationships in their life” [2]. Personal relationships can be fur- 
ther described as a network that is created through the close connections one main- 
tains with others and that implies an emotional investment of some sort [3]. Personal 
relationships are also highly mutable and are impacted daily by one’s immediate 
social environment [4]. While the perceived quality of one’s relationships influences 
QoL to a significant degree, it is also important to note that the quality of one’s 
social interactions (and not the quantity) is the best predictor of self-reported health 
[5], like the relationships themselves, the perceived quality of relationships is also 
susceptible to change. While the perceived quality of one’s relationships influences 
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QoL to a significant degree and is dependent upon numerous factors. Social 
exchange theory, for example, argues that the perceived quality of any relationship 
depends on the (1) the perceived benefit vs. cost ratio (BCR); (2) the individual's 
comparison level (CL), or what they believe they deserve to get out of a particular 
relationship in terms of BCR; and lastly, (3) the individual's comparison level for 
alternative (CLalt), which is the lowest BCR outcome they are willing to accept in 
a specific relationship given the perceived BCR outcomes of alternative relation- 
ships or of merely being alone [6]. Furthermore, equity theory suggests that BCR 
outcomes determine the quality of a personal relationship to the extent that they 
affect the relationship's perceived fairness [7]. 

Research on QoL and personal relationships, along with other complex social 
Science issues, has until now been burdened and held back by technological limita- 
tions. Researchers of these topics, for instance, have generally accepted a reliance 
on self-reported data from small sample sizes. Additionally, due to ethical and 
financial limitations, collection techniques such as mobility traces and the use of 
personal call records have been relatively rare. Such limitations have had particu- 
larly adverse implications for intricate longitudinal research designs, making it dif- 
ficult for researchers to engage individual participants over time. The number of 
iterations or data points researchers have been able to study has also been limited, 
as methodologies based on pen-and-paper self-reporting were costly and made the 
logging of daily, weekly, or even monthly changes in behaviors and states procedur- 
ally burdensome. 

However, with the advent of computers, the Internet, and, most importantly, the 
nearly universal adoption of smartphones, a more suitable way to record such data 
emerged. Unlike traditional survey instruments, modern devices such as smart- 
phones, smartwatches, and other wearables allow researchers to monitor behaviors 
and social interactions effortlessly and without additional engagement from partici- 
pants. They have also allowed further innovations in the measurement of personal 
relationships by enabling real-time and synchronous data recording. Moreover, 
developments in modern data science, including advancements in sensor networks, 
machine learning, deep learning, and AI, offer novel ways of capturing and analyz- 
ing observed phenomena. In accordance with General Data Protection Regulation 
(GDPR) guidelines, researchers can accumulate large amounts of private data, such 
as location information, call logs, and messages (from SMS and applications such 
as Facebook Messenger, WhatsApp, and Viber), provided they first obtain partici- 
pants’ permission, which has permitted novel insights into a range of scientific 
inquiries. Cloud technologies, in turn, allow for almost unlimited data storage, 
which removes the limitations on sample size that troubled previous research. 
Finally, the global availability of smartphones allows sample representativeness in 
relevant populations to be more easily attained, thereby ensuring studies' external 
validity. Some pioneering researchers have already applied algorithms to data per- 
taining to mobile phone usage in order to measure different facets of communica- 
tion, such as the frequency and duration of virtual communications and their 
relationship to tie strength [8]. 
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Given these developments, this chapter aims, first, to explore the methodologies 
that have been used until now in studies focusing on the assessment of QoL and 
personal relationships, indicating, where relevant, their deficiencies and shortcom- 
ings; and, second, to propose a methodology that overcomes these shortcomings by 
using new digital technologies to capture and analyze relevant data in a reliable and 
timely manner. 


Assessment of Personal Relationships and Their 
Impact on QoL 


In studying personal relationships and their relation to QoL, researchers have relied 
on a variety of existing validated QoL measures that include assessments of the 
quantity, quality, and intensity of personal relationships to quantify QoL. As a vari- 
able, personal relationships can encompass a diverse range of phenomena, poten- 
tially including everything from marital relationship, kinship, and friendship to 
neighborhood-based relationships and acquaintances with fellow members of a 
church or club. Some studies, for instance, have considered the concept of relation- 
ship to equally imply both sexual and private relations, hence overlapping personal 
relationships with relationships of sexual intimacy (as covered in the WHOQOL 
facet "Sexual Activity" [2]). In some studies, personal relationship measures have 
overlapped with measures from the WHOQOL "Social Support" facet, which 
focuses more on the social structures an individual belongs to and the social support 
that is available to them when needed than personal relationships in general [2]. 

Below we provide a list of psychometrically standardized and validated QoL 
measures that involve some assessment of personal relationships as a variable. 
Taken together, these measures (a) apply to a range of demographics, including ages 
ranging from childhood to adulthood; (b) take various operationalizations of per- 
sonal relationships into account, ranging from holistic to multi-faceted approaches, 
and (c) address various socio-environmental contexts, ranging from a particular 
context to a collective context, such as family. 


1. WHOQOL-100 [9]. 

The World Health Organization has made sure to develope and to publish an 
instrument that measures quality of life in a quantitative manner. It consists of 
100 items and therefore is titled WHOQOL- 100. This instrument produces vari- 
ous scores, namely related to (a) particular facets of quality of life (e.g., social 
support and financial resources); (b) larger domains (e.g., physical, psychologi- 
cal and social relationship domain), and (c) overall quality of life and general 
health. With regards to personal relationships, WHOQOL-100 contains items 
such as “How satisfied are you with your personal relationships?" Furthermore, 
WHOQOL-100 is both culturally sensitive and psychometrically sound. 
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2. KIDSCREEN-27 [10-13]. 

The KIDSCREEN project involved 13 European countries in the develop- 
ment of a cross-culturally harmonized QoL measure designed to be administered 
to children. The resulting KIDSCREEN instrument was based on literature 
reviews, expert consultation, and focus groups conducted in all the participating 
countries. The pilot version consisted of 185 items, which, after elimination of 
some items, was reduced to 52 of the original items. Later, a shorter, 27-item 
version of the same measure, called “KIDSCREEN-27,” was created and vali- 
dated [14]. Concerning personal relationships and their relation to QoL, 
KIDSCREEN-27 assesses the quality of personal relationships a child can 
develop in school. Items such as *Have you spent time with your friends?" 
“Have you had fun with your friends?” and “Have you and your friends helped 
each other?" are intended to evaluate the safety net of personal relationships that 
serve as building blocks in the development of one's self-esteem. 

3. Satisfaction with Life Scale (SWLS) [15]. 

A widely used and validated measure [16-18], the SWLS encompasses five 
broad, global items and allows respondents to weight domains of their lives 
according to their own beliefs and values. The five-item scale provides a holistic 
and overarching assessment of an individual's satisfaction with their life. In the 
area of personal relationships, the SWLS's measure of QoL takes into account 
items such as “In most ways, my life is close to my ideal,” “The conditions of my 
life are excellent,’ and “I am satisfied with my life,’ to which participants 
respond with a rating from 1 (strongly disagree) to 7 (strongly agree). Although 
the items do not reference personal relationships directly, individuals with close, 
fulfilling relationships are much more likely to report being satisfied with their 
lives or describing the conditions of their life as excellent or ideal. Assessments 
of the quality of one's relationships can therefore be a useful predictor of an 
individual's ratings for these three items. 

4. RAND-36 [19-21]. 

This measure has been validated in multiple studies [22, 23] and represents 
one of the most common measures of health-related QoL (HRQoL). It generates 
ordinal level data for each of its 36 items, which are in turn aggregated to sub- 
scale scores [22]. When assessing the quality of life with regards to personal 
relationships, RAND-36 limits the time frame of the retrospection. This can be 
shown in the following item: “During the past 4 weeks, to what extent has your 
physical health or emotional problems interfered with your normal social activi- 
ties with family, friends, neighbors or groups.” The participants are then required 
to choose between responses varying from “all of the time" to “none of the time." 

5. Beach Center Family Quality of Life (FQOL) Scale [24—26]. 

The FQOL scale is an inventory consisting of 25 items rated on a five-point 
Likert-type scale that has been widely used and validated. It measures satisfac- 
tion with family life as a significant factor in QoL. In particular, it assesses fam- 
ily interaction, parenting, emotional well-being, physical/material well-being, 
and disability-related support [24] and contains items such as “My family mem- 
bers talk openly with each other," “My family is able to handle life's ups and 
downs,” and “My family enjoys spending time together.” 


14 Beyond Pen and Paper: Reimagining Assessment of Personal Relationships... 359 


Apart from these validated measures, some researchers have leveraged non- 
standardized instruments and interviews to assess QoL, while others have applied 
more innovative approaches, such as extrapolating QoL measures from tests of 
social cognition and self-reported measures of social understanding [27]. However, 
as these approaches are far from widely adopted and often do not consider social 
relationships, we do not elaborate on these in this chapter. 


Assessment of Personal Relationships Via Digital 
Item Representation 


Having discussed the most common approaches to assessing personal relationships 
in measures of QoL, we now outline ways in which the items included in these 
measures have been assessed using digital technologies. We define digital item rep- 
resentation as the digitalization of any item of a particular instrument that maintains 
the format in which the item was traditionally recorded (such as binary or Likert 
scale formats). For an item assessing how much time an individual spent with their 
spouse in the past week, for example, this might involve the collection and record- 
ing of relevant information via GPS, Cell-ID positioning, WiFi information, or 
Bluetooth signals acquired directly from both spouses' smartphones and identifying 
the intervals when the two devices were in close proximity, thus rendering the indi- 
vidual's self-reporting superfluous. The automation that such techniques may 
increase engagement among study participants and motivate them to provide more 
data while also reducing dropout rates. It could also be argued that the quantitative 
and objective data derived from GPS, Cell-ID, WiFi, and Bluetooth technologies is 
likely to be a more accurate source of information than participants' subjective rec- 
ollections [28, 29]. Such a position is further supported by research that has demon- 
strated that the context, wording, and format of questions can significantly impact 
the responses one receives [30]. Leveraging quantitative and objective datasets in 
the manner described above allows one to avoid the introduction of error from these 
sources. 

An example of how digital techniques can be used to investigate social relation- 
ships is provided in a study conducted by Wiese et al. [8], where the researchers 
used a computational model to assess and classify personal relationships. Using call 
and text message logs from smartphones as inputs, Wiese et al. leveraged an algo- 
rithm to classify participants’ contacts into three relationship categories: family, 
work, and social relationships. Based on extracted features such as the intensity, 
regularity, duration, and medium of communication, the authors were able to clas- 
sify relationship categories with up to 90.596 accuracy [8]. Meanwhile, an ambi- 
tious approach to relationship imaging is the social MRI method developed by 
Aharony et al. [31]. Using credit card records and information on social media and 
mobile phone use, including calls and text messaging, this method provides an 
objective means of visualizing social systems. The authors also utilized self- 
reporting to collect Big Five personality tests and assessments of participants' 
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momentary moods and sleep quality, which were later used to detect correlations 
between these variables and aspects of participants’ social relationships. By incor- 
porating vast amounts of privacy-sensitive data in their model's framework, the 
authors were able to "help further our understanding of the interconnections and 
mechanics of human society" [31]. 


From Paper-Based Surveys to Digital Item Representation 


Every instrument mentioned in section "Assessment of Personal Relationships Via 
Digital Item Representation" operationalizes a specific variable or set of variables 
corresponding to a specific item in terms of data that can be collected from digital 
devices. Incorporating digital item representation is a process that involves replac- 
ing traditional data recording methods (in particular, the method of self-reporting) 
with more technologically savvy ones. An initial step towards digital item represen- 
tation is made with the use of electronic patient-reported outcomes (ePRO; [32]), 
which adapts traditional, paper-based means of assessing patient states as patient- 
reported outcomes (PROs; [33]) to a digital format so that patients can complete 
them electronically. The next step in implementing digital item representation is to 
replace the self-reported data sources, which are memory-based, subjective, and 
infrequent, with quantitative context- and sensor-based sources that are objective 
and frequent. In what follows, we discuss the latter of these steps, focusing in par- 
ticular on the ways digital item representation might be applied to the QoL mea- 
sures described in section "Assessment of Personal Relationships and Their Impact 
on QoL” in order to better quantify the aspects of personal relationships that influ- 
ence QoL. 

To illustrate how digital item representation could be applied in the case of the 
WHOQOL facets, we take as an example a study by Chang et al. [34] that explored 
the mediating effect of depression on WHOQOL facets such as positive feeling 
and social support. In this study, depression was measured via traditional paper- 
based self-reporting. To assess each individual's depression and the quality of their 
personal relationships, the researchers used a WHOQOL-100 item “How alone do 
you feel in your life?" Another WHOQOL-100 item that would also be suitable for 
digital representation is “How satisfied are you with your personal 
relationships?" 

In this case, the researchers, along with the healthcare practitioners caring for the 
patient, might have benefited from having additional, more objective data sources 
regarding the intensity and frequency of the patient's social interactions. It has been 
demonstrated, for instance, that depressive patients tend to suffer cognitive distor- 
tions that lead them to downplay positive events in their life, which further exacer- 
bates feelings of depression [35, 36] and could lead to inaccurate self-reporting. As 
a step towards digital item representation, collecting GPS, Cell-ID, WiFi, and/or 
Bluetooth data from the patient would have allowed the researchers to objectively 
measure the time the patient spent at home or close to those whom they recognize 
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as their significant others. An individual’s close friends and contacts could be recog- 
nized by training an algorithm to assess the intensity and frequency of communica- 
tions with specific individuals as well as analyze the individual’s social media 
profiles and use of social media platforms [37]. This approach could provide a reli- 
able way of evaluating the quality of individuals’ relationships that is more accurate 
than asking them directly, leading to more objective assessments of the WHOQOL- 
BREF facet relating to personal relationships, which asks “How satisfied are you 
with your personal relationships?". By improving the accuracy of information, it 
would also allow both practitioners and researchers to better distinguish causal pat- 
terns between aspects of personal relationships and QoL. Finally, in the case of 
depression, as studied by Chang et al., the use of these measurement techniques 
could significantly impact researchers’ or practitioners’ assessments regarding the 
state of an individual's depression, the cause of their depression, and the potential 
mediating impact of personal relationships on their depression. 

The KIDSCREEN-27 measure, meanwhile, includes a facet called “Peers and 
Social Support" [38]. The traditional paper-based KIDSCREEN-27 tries to opera- 
tionalize this facet via items such as *Have you and your friends helped each other?" 
[11]. However, a digital representation of this item would be possible via an algo- 
rithm that incorporates sociometric techniques to render a sociogram, which would 
in turn provide information about specific peer dynamics inside an observed cohort 
of children [39]. To develop a clear, unbiased picture of this item, this algorithm 
could monitor the social network activity of a child (including information such as 
GPS data) and analyze the online content they are posting, sharing, liking, or com- 
menting on (e.g., on their phone, tablet, or computer), as well as other relevant 
information such as the number of hours the child spends playing outside or in 
multiplayer games online. 

As for the Satisfaction with Life Scale (SWLS; [15]), traditional items that ask 
participants to retrospectively assess and self-report their satisfaction with their 
lives could easily be digitalized via a mobile application that would occasionally 
prompt the individual to engage in a brief QoL assessment on their phones. Such an 
assessment would leverage methods such as experience sampling methods (ESM; 
[40, 41]), also referred to as ecological momentary assessment (EMA; [42]), and be 
designed to capture momentary, self-reported ratings of experiences, moods, 
thoughts, symptoms, or behaviors that are expected to change over time. Such 
momentary methods have been shown to be psychometrically superior to the usual, 
retrospective self-reports with their longer recall periods [43]. The near-ubiquitous 
availability and affordability of smartphones has contributed significantly to the 
realization of the ESM/EMA methodology’s true potential by facilitating the cap- 
ture of momentary data. Not only do individuals in most developed countries already 
possess this potential research hardware [44], but they also carry it at their side 
throughout the day [45], providing researchers with a potentially unlimited stream 
of self-reported as well as passive datasets. Despite the benefits of conducting self- 
reporting with a smartphone, however, there is still room for improvement. In a 
comprehensive review of the use of mobile phone devices in ESM, Van Berkel et al. 
[46], have noted that the notifications prompting participants to rate their feelings 
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are often burdensome. These prompts can be adapted by analyzing user-generated 
data to send the prompts at times when the individual is not occupied with other tasks. 

While self-reports provide important information on an individual's situational 
contexts, they continue to rely on the user and require mobile notifications that are 
likely to continue being somewhat disruptive. As described above, data concerning 
one's situational context can be collected passively by leveraging multiple sensors 
that are embedded within smartphones by default [47]. For example, an automatic, 
continuously sensing smartphone-based application may incorporate data from a 
phone's accelerometer, microphone, light sensor, GPS, Cell-ID, WiFi, and Bluetooth 
activity. Data collected through smartphone sensing—including call logs, mes- 
sages, and data on sleep, physical activity, and location—could also provide infor- 
mation about individuals’ social interactions, activities, and mobility [48]. It is 
further possible to create a computational model that leverages deep learning tech- 
nologies for classification tasks on data received from sources such as GPS, Cell-ID, 
WLAN, and call, SMS and social media logs. In summary, the techniques of digital 
item representation discussed above, including the use of ESM/EMA-based self- 
reporting and the collection of passive datasets using smartphones, could be lever- 
aged for SWLS assessments in individuals' daily lives. Using smartphones for 
momentary self-reporting can improve the reliability of individuals' responses to 
items such as “In most ways, my life is close to my ideal," “The conditions of my 
life are excellent,’ and “I am satisfied with my life.’ Meanwhile, computational 
models of individuals' social, physical, and recreational activity developed through 
passively collected data can help researchers and practitioners better assess the 
veracity of responses and the factors that are correlated with positive or negative 
assessments. 

Another instrument mentioned above that is used for assessing personal relation- 
ships is the RAND-36 [22]. Apart from asking participants to report the same infor- 
mation via a digital platform and thus leveraging the ESM/EMA approach, there are 
specific methods that can be implemented to complement self-reported RAND-36 
data. In section "Assessment of Personal Relationships and Their Impact on QoL” 
we have already emphasized a specific RAND-36 item pertaining to personal rela- 
tionships (“During the past 4 weeks, to what extent has your physical health or 
emotional problems interfered with your normal social activities with family, 
friends, neighbors or groups"). The scope of this item is fairly broad and we can 
therefore measure it in various ways. Firstly, it can be assessed through analysis of 
the specific patterns or content of one's conversations. Conversations can be logged 
to detect patterns, while text collected from messaging apps and SMS logs can be 
semantically analyzed. An example of how an algorithm can be deployed to analyze 
textual output on social media is found in the recent case of Weibo users being 
monitored amidst the COVID crisis in China. The algorithm identified persons at 
risk of suicide and alerted responsible volunteers to contact authorities [49]. 
Secondly, to assess the tangible element of RAND-36's social activities, an algo- 
rithm that analyzes textual output could be implemented in combination with GPS, 
Cell-ID, WiFi, and Bluetooth proximity data to complement self-reported responses. 
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Thirdly, the emotional element of social activities could be further evaluated based 
on analysis of the emotional content and intensity of virtual interactions found in 
messaging app and SMS logs, including the use of emoticons. Self-reporting con- 
cerning the facet of positive social interactions could be complemented with GPS, 
Cell-ID, WiFi, and Bluetooth data that indicates an individual’s most frequently 
visited places (via geolocation) and time spent in proximity to other individuals (via 
analysis of other devices in one’s proximity). 

Finally, in the case of the Beach Center FOOL [25], which contains items such 
as “My family members talk openly with each other,’ “My family is able to handle 
life’s ups and downs,” and “My family enjoys spending time together,” digital item 
representation could be carried out using GPS, Cell-ID, WiFi, and Bluetooth prox- 
imity data for the individual and their family members. Logging the locations where 
family members spend time together and the duration of their interactions could 
provide insight into familial social relationships and enrich the data gathered from 
self-reporting. Furthermore the call, SMS, and messaging app logs of family mem- 
bers might be compared to assess individual family members' communication 
styles. Multiple communication parameters could be recorded, such as frequency, 
duration, content, expression style (analyzed for emotional expressions), and most 
preferred recipient. If a family collectively uses a messaging app such as Viber or 
WhatsApp, a group chat would offer valuable data for assessing individual family 
members' communication styles while exploring family dynamics. 

A summary of suggestions for implementing digital item representation in the 
QoL measures discussed above is presented in Table 14.1. 

Researchers seeking to assess QoL using QoL measures such as those mentioned 
above can enrich their methodological toolboxes by making use of accurate, timely, 
and privacy-conscious computational models of personal relationships developed 
using the techniques mentioned above. Since many researchers continue to use self- 
report-based, non-standardized, single-item QoL measures that are conceptually 
broad and fail to establish structural relations between variables or provide in-depth 
insights (e.g., “Describe your quality of life"; [27, 50-52]), in the following section 
we discuss the necessity for further research to establish the validity of digital item 
representations. 


Discussion: Limitations of Digital Item Representation 


As discussed above, digital assessment via smartphones and wearables can provide 
researchers with larger sample sizes and render more accurate, synchronous mea- 
surements. Problems relating to sample size, representativeness, external validity, 
and assessment standardization could be singlehandedly solved by using validated 
digital platforms to mine and store data. An important issue that emerges as research- 
ers move towards collecting data from applications and other smartphone metrics, 
however, is privacy. There is rising concern among mobile phone users regarding 
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Table 14.1 Proposed digital item representations of items from selected QoL questionnaires 


Instrument Items Sources to quantify the factor 

WHOQOL-100 |“How alone do you feel in your life?"; Smartphone (GPS, Cell-ID, 
“How satisfied are you with your personal | WLAN, data use info for indoor/ 
relationships?" outdoor activity assessment; 

call, SMS, social media 
messenger logs); 
ESM/EMA (for loneliness 
assessment) 

KIDSCREEN-27 | “Have you and your friends helped each | Smartphone (calls, social media 
other?" activity); gaming consoles 

(gameplay logs); personal 
computers (eLearning platforms, 
gameplay logs) 

SWLS “How would you rate your satisfaction Smartphone (call, SMS, social 
with your own life in the past three media messenger logs); 
months?"; “How often do you feel sad?" | ESM/EMA (for sadness 

assessment) 

RAND-36 | “During the past 4 weeks, to what extent | Smartphone (GPS, Cell-ID, 
has your physical health or emotional WLAN info; call, SMS, social 
problems interfered with your normal media messenger logs); 
social activities with family, friends, ESM/EMA (for quality of 
neighbors or groups.” personal relationship rating 

assessment) 

Beach Center “My family enjoys spending time Smartphone (GPS, Cell-ID, 

FQOL together”; “My family members talk WLAN info; call, SMS, social 
openly with each other.” media messenger logs for family 

members) 


privacy, data sharing within applications, and security breaches [53-55], the latter 
of which allow unwanted parties to collect certain information about users and use 
it for marketing or more sinister purposes. For this reason, the European Union 
introduced the General Data Protection Regulation (GDPR) in 2018, which focuses 
on data protection and privacy limitations and outlines necessary data protection 
practices in the European Union. These regulations thus specify legal and ethical 
limitations to the data practices researchers in Europe can adopt to complement 
standard survey practices. As regulations in distinct areas are likely to differ, the 
ways in which digital research methodologies can be implemented will vary greatly 
by geographic region. 

Furthermore, with the use of data-gathering applications and algorithms, elabo- 
rate consent forms and data security protocols need to be implemented to protect 
participants’ privacy. Rather than study the relevant regulations and implement 
these protocols, many researchers are likely to opt out of using such tools despite 
their benefits. However, initial steps have been made towards implementing these 
methods more broadly, while several studies have been able to evaluate users’ atti- 
tudes towards data sharing, providing future researchers with concise and specific 
suggestions concerning how to increase the validity and trustworthiness of their 


14 Beyond Pen and Paper: Reimagining Assessment of Personal Relationships... 365 


digital instruments [56]. When collecting smartphone, wearable, or EMA data from 
users in order to assess their relationships, one essential requirement is that research- 
ers be clear about the immediate aims of their research, as well as the fact that their 
ultimate goal is to improve the individual’s life quality in the long term. As far as 
child privacy is concerned, when using instruments such as the KIDSCREEN-27, it 
is of the utmost importance that researchers acquire parental permission before col- 
lecting any data. 

The scope of the research presented in this chapter has had certain limitations. 
We do not offer a systematic literature review of all the instruments used to evaluate 
personal relationships in relation to QoL, nor is our discussion of the possibilities 
offered by smartphones and wearables for the digital quantification of the above 
items comprehensive, as such devices as smartphone gyroscopes and wearable heart 
rate and galvanic skin response monitors likely present additional research applica- 
tions. Nevertheless, this chapter has opened a discussion on the digitalization of 
current methods of assessing personal relationship as a factor in ratings of QoL. 

Looking towards the future, recent research results have shown that the use of 
new, personal, miniaturized technologies is bound to become prevalent. Indeed, 
there are already individuals who are taking part in the “quantified self’ movement, 
leveraging these technologies for assessment of their own personal relationships 
[57]. One noteworthy wearable device in this trend is the Filip Smartwatch (https:// 
www.myfilip.com/), which supports family communication and location-based 
information exchange without the use of a smartphone. Developments in the quanti- 
fied self community are at the forefront of the trend of “self-knowledge through 
numbers,” and personal relationships are just one of the relevant areas individuals 
seek to quantify and improve. Overall, technologies used to assess personal rela- 
tionships are examples of quality of life technologies (QoLT), a term referring to 
any technologies used for assessing or improving an individual’s QoL. These tech- 
nologies leverage the increasing availability of miniaturized, communication sen- 
sor- and actuator-based, context-rich technologies for computation and storage that 
can be embedded within personal devices such as smartphones and wearables [58]. 
There is therefore a promising future in the use of QoLT for the assessment of per- 
sonal relationships based on data collected from daily life environments. 


Concluding Remarks 


With the further implementation and standardization of digital item representations 
for the assessment of personal relationships and QoL, researchers and practitioners 
will have excellent opportunities develop more accurate and timely knowledge 
about their study participants and patients. As this chapter has discussed, smart- 
phones and wearables can be utilized to perform standardized momentary QoL 
assessments, which can be administered to individuals through a simple and user- 
friendly mobile interface. At the same time, the fact that the methods described in 
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this chapter are unobtrusive provides researchers the opportunity to obtain a more 
nuanced longitudinal, context-based view of individuals' personal relationships and 
QoL and to identify behavioral correlations. Leveraging both self-assessments and 
passive datasets, relationship models could be developed that provide researchers 
with more information while measuring an individual's states and behaviors in real- 
time and in their present context, allowing for better assessment of questions such 
as “How alone do you feel in your life?" Finally, similar to the use of standardized 
tests, objectively acquired data would enable behavioral and computer scientists 
access to quantitative data that is standardized and comparable, permitting the 
development of algorithms that provide further insight into the connections between 
aspects of personal relationships and QoL. Implementing these technologies in new 
ways can thus lead to new improvements in individuals' everyday lives. 
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Introduction 


Our friends and families define the fabric of our lives, and the supportive network 
they construct determines much of well-being. Beyond (or perhaps due to) feelings 
of warmth and belonging, strong social support is related to lower rates of morbid- 
ity, mortality, and to better cardiovascular, neuroendocrine, and immune function 
[1]. The quantity, quality, structure and function of these intricate and dynamic net- 
works is a critical component of quality of life and should be considered carefully 
when assessing and improving health. Social support is the provision of psychologi- 
cal and material resources from one's social network intended to benefit an indi- 
vidual [2]. There are several ways in which researchers have defined and 
operationalized social support, but House and colleagues' theoretical framework 
about the domain of social support is particularly clear and comprehensive [3]. This 
framework delineates three approaches for understanding the components of social 
support and aligns well with various assessment techniques. 

The simplest and most direct method of assessing social support is to examine 
the quantity of social support. Measuring marital status, number of friends, and 
community involvement (e.g., church membership) are the most common variables 
that quantify this concept. Terminology such as social integration, isolation, loneli- 
ness, and social embeddedness are used to describe this facet of social support [4]. 
This type of approach to quantifying social support is relatively objective, reliable 
and easy to obtain [5]. The Social Network Index (SNI [6]) is the most comprehen- 
sive and popular tool for measuring the quantity of social relationships. The SNI is 
a self-reported questionnaire that quantifies social connections, evaluates the 
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frequency of contact, and categorizes individuals on a continuum from socially iso- 
lated to socially integrated. The SNI measures marital status, sociability (how many 
friends and family members one has and how often they are in contact), participa- 
tion in a religious group, and participation in other community groups. Some mea- 
sures, such as the Social Support Questionnaire [7] add another element by assessing 
both the number of people respondents feel they can count on as sources of social 
support and their satisfaction with the support they receive from each those people. 
While this straightforward approach for assessing the existence of social connec- 
tions is useful and informative, it does not encapsulate the full complexity and varia- 
tion of social support. 

Social support may also be assessed by moving beyond the mere number of 
social connections and focusing on the structure of relationships [3]. A social net- 
work analysis approach broadens the range of relationships considered, includes 
both positive and negative influences of relationships, and analyzes the patterns of 
relationship structure [8]. This approach provides a richer level of detail because it 
assumes that not all relationships are created equal, and different facets of social 
connection may have differential impacts on well-being. There are several charac- 
teristics of social networks (e.g., size, density, reciprocity, homogeneity) that should 
be considered when assessing these social connections [9]. This approach provides 
detailed insight into the interconnected webs of social relationships and how mul- 
tiple levels of influence may impact an individual. In the past, this methodology has 
been limited by a lack of data. However, with the advent of the internet and big data 
approaches, social network analysis has boomed in recent years. 

If one hopes to pinpoint the specific effects of social support on well-being, it 
may be advisable to adopt a functional approach that emphasizes the differing influ- 
ence of various types of support [3]. While quasi-objective support measures exam- 
ining the mere number of connections or structure of those connections are valuable 
[10], it is important to look at whether an individual believes that enough support 
exists to help them in times of need. The type of support provided, the source of 
support, and the manner in which it is delivered are key aspects that determine the 
effects of social support on well-being [11]. One type of social support, emotional 
support, entails sympathy and love, encouragement, communication or care that 
may reduce negative psychological states. Informational support is characterized by 
advice, facts or information that may assist in addressing a problem. Instrumental 
support is providing tangible assistance such as money, resources or time [12, 13]. 
Each of these types of support serves its own unique function and it may be infor- 
mative to delineate the differences when assessing health-relevant impacts. 

Many of the most robust and informative social support scales are functional in 
nature. The Interpersonal Support Evaluation List (ISEL [13]) is the most widely 
used instrument to assess perceptions and functions of support. This is particularly 
important because the perception or appraisal of that support is the key element that 
drives many of the positive health effects found in the literature [4]. Important social 
support measures also include the Multidimensional Scale of Perceived Social 
Support [14] (MSPSS), a 12-item instrumental and emotional support scale that 
measures the degree to which respondents feel or perceive that they are supported 
by friends, family or a significant other. Questions used to assess this include “J 
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have a special person who is a real source of comfort to me", "I can talk about my 
problems with my family” and “I can count on my friends when things go wrong” 
to which respondents rate each statement on a 7-point likert scale ranging from 
1 = Very Strongly Disagree to 7 = Very Strongly Agree. Similarly, the Duke-UNC 
Functional Social Support Questionnaire [15] assesses the amount of support 
respondents receive and categorizes this support into affective, confidant, and 
instrumental categories. This is an instrument frequently used by medical profes- 
sionals since higher levels of social support are often tied to better medical adher- 
ence. Questions are rated on a 5-point scale ranging from | = “Much less than I 
would like" to 5 = “As much as I would like" and include items such as, “J get help 
when I am sick in bed" and "I get useful advice about important things in my life". 
Researchers assert that functional measures are more informative because they 
target the specific influences provided by one's social connections [16]. For exam- 
ple, adequate functional support may be derived from one very good relationship, 
but may not be available to those with multiple superficial relationships. This is 
particularly important to consider in the digital age because we have so many plat- 
forms of communication and many “weak” online relationships. A functional oper- 
ationalization helps to clarify the psychosocial impacts of the many 
constantly-evolving ways in which people exchange support online. 


How Does Social Support Influence Health? 


Provision and perception of social support has been linked to a variety of positive 
well-being outcomes, particularly in the realm of health [17]. Social support pre- 
dicts physical health [18] and each type of support (instrumental, informational, 
emotional) offers a unique blend of benefits [19]. Social support serves as a protec- 
tive factor against stress and chronic illness, and confers numerous benefits on an 
individual's psychological and physiological well-being [1]. However, it is impor- 
tant that the type of support properly aligns with the needs of the individual. If the 
type of support offered matches the type of support desired, then it will most likely 
lead to positive health outcomes [11]. 

How does this support actually influence well-being and health? One theory is 
that social support influences health outcomes primarily through its ability to buffer 
stress. A strong social support system may reduce the negative effects of stressful 
experiences by providing a less threatening interpretation of a stressor and allowing 
the individual to feel that they have the proper resources to cope with the situation 
[2]. Since high levels of stress are linked to negative physical health outcomes (e.g., 
allostatic load), reduction of stress through reliance on social support systems may 
ultimately be beneficial for a variety of health outcomes. Another theory posits that 
social support may be beneficial in all situations, regardless of whether stress is 
involved. This main-effect theory states that social integration may directly influ- 
ence health through things such as social control and normative behavior [16]. Most 
likely these theories operate in tandem, and both partially explain how social sup- 
port may promote positive outcomes. 
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Assessing and Conceptualizing Social Support in the Digital Age 


The advent of the internet and smartphone culture has fundamentally transformed 
the nature of social support. Technological change has altered (1) The ways in 
which we assess social support, (2) The perception and effects of social support. In 
the following chapter, we will discuss these two areas at the intersection of technol- 
ogy, social support, and health. 

In the first part, we will examine how recent technological innovations have 
allowed for much more detailed, objective, and accurate assessments of social sup- 
port. A large portion of one’s social interactions are conducted online, and since 
online activity can be tracked and analyzed, we are able to peer into the window of 
one’s life and gain a better understanding of how their social relationships unfold. 
In addition, we have developed tools that allow us to capture more fine-grained and 
real-time data about social interactions that can better inform our understanding of 
in-vivo social connections. 

In the second part, we will discuss how the concept of social support has changed 
in the age of digital communication. We will focus on how the presence and use of 
technological devices influences face-to-face interactions, online groups, and fam- 
ily dynamics. To conclude the chapter, we will summarize the current research from 
a variety of domains about the assessment of social support via digital technology. 
We will identify gaps in the literature, challenges for researchers and practitioners, 
and important areas for future directions. Taken together, this chapter will recognize 
the changes in social assessment afforded by technology and consider several 
important areas in which technological tools have transformed social support. 


How Has Technology Altered Our Assessment 
of Social Support? 


New technologies represent an immense opportunity for clinicians and researchers 
to study social support's links to well-being and health, as well as the dynamic 
social interactions that underpin these relationships. Researchers today have access 
to more data than ever before. The potential for gaining theoretical and actionable 
insights abound. By leveraging big data across several accessible technological 
platforms, researchers can begin to understand how social support processes unfold 
in real time and the myriad ways technology can be used to measure meaningful 
aspects of social support. 


But First: What Are Big Data? 


In the social sciences, big data typically represent large-scale data comprised of 
many thousands of people and sometimes hundreds of data points about any single 
person. In essence, these data are long and wide. A single row could represent one 
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person, or sometimes, a single measurement of one person among thousands of 
other people. Big data are typically longitudinal, and the granularity of measure- 
ment depends on the source from which the data come. Although big data manifest 
in many forms, there are two types that may be particularly useful for studying 
social support. The first is to use big social media data scraped from platforms like 
Twitter (est. 2006, 330 million users worldwide) or Facebook (est. 2004, 2.6 billion 
users worldwide). These data are longitudinal, naturalistic, and involve millions of 
social interactions, by design. The other is via big personal data captured on a 
device with which most people in the world are now intimately familiar: the 
smartphone. 


Big Social Media Data 


Harvesting data from social media platforms to learn about social support in the 
wild makes good sense. These platforms are social by design. We “friend” and “fol- 
low” people we know (or are attracted to or have interest in). Using features of these 
platforms, we interact with people in our close circles and with people who are far 
outside the reaches of our immediate friend group, sometimes with complete strang- 
ers across the country or the world. We post messages and media to our timelines 
and walls, sometimes making those posts searchable and visible to anyone who is 
willing to attend to them. We pour ourselves into a networked digital social sphere, 
sometimes hoping for connection, sometimes seeking advice, and in any case, 
because we believe someone will read or listen to the content we post and respond 
in kind. Because of the accessibility of their data, most researchers study user posts 
and interactions on Facebook and Twitter. Below, we offer a flavor of possible ways 
to use data from these platforms to study social support. 


Facebook 


Facebook is a platform for personal online diary-like posts, containing text, images, 
and GIFs, organized in timelines, where some of the content is strictly private and 
by default is shared only with members of an individual's social network of approved 
friends. In some cases, personal profile posts can be shared publicly such that any- 
one in the world who navigates to a user's profile can see public content. One criti- 
cal feature of the Facebook platform is the ability to form groups based on common 
interests and experiences. For example, cancer survivors who use Facebook can 
search the platform for existing groups of other survivors and join those groups to 
connect with strangers who have experienced similar struggles with fighting their 
particular type of cancer. If the group is designated as public, a researcher can scrape 
posts and replies on the group's main page to evaluate the provision of social sup- 
port in this context. Analyzing the content of posts makes it possible to differentiate 
the provision of emotional, informational, or instrumental support across users, 
explore the dynamics of support provision over time, and quantitatively analyze the 
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popularity of posts to gauge which types of support are most valued in the group. 
Moreover, there are tens of thousands of groups on Facebook. This provides a 
unique opportunity to determine how the provision of social support varies by con- 
text. Extending this example a bit further, it would be possible to explore how can- 
cer survivors from different groups across the country provide social support; 
alternatively, one could compare the provision of social support among groups 
formed around other health issues, identities, social statuses, political ideolo- 
gies, etc. 

Although there are many opportunities to study the provision of social sup- 
port on Facebook, there are some important drawbacks to consider. The first is 
that researchers typically cannot link the receipt of social support from group 
members to psychological or health outcomes associated with any single user. 
For example, if a user posted a message about seeking prayers or advice for a 
loved one in the hospital, there is no way to link the level of emotional or infor- 
mational support expressed in responses to this post to any outcomes related to 
the original poster (there are exceptions to this that are not worth mentioning 
here). The second is that it is difficult to get general user profile information 
(e.g., Facebook timeline posts) that allow for analyses of long-term outcomes. 
For example, if support was provided in one moment on a public Facebook page, 
researchers cannot evaluate how that provision of support predicts a user’s posi- 
tivity in their posts to their personal timeline 6 months later. Without explicit 
permission from the Facebook user, no access to a personal timeline is granted. 
Thus, Facebook data are excellent for understanding how groups of users inter- 
act with one another to provide support to each other through the messages they 
post. These data present an opportunity to unpack underlying processes in sup- 
port provision, but unfortunately do not allow us to easily link this provision to 
important outcomes. 


Twitter 


Twitter is a social media platform on which users can create micro-diary posts 
(i.e., tweets) limited to 288 characters but can contain pictures, videos, and links. 
By default, Twitter profiles are public, meaning that anyone who accesses a pro- 
file can see a user’s posts. If the user desires, they can convert their profile to 
private such that only approved followers have access to a timeline content. The 
ubiquity of Twitter data makes it possible to link support provision with indica- 
tors of psychological well-being. This is the case because Twitter data are com- 
prised of user profile information and tweet generated over time. Thus, they are 
more flexible for not only describing social support provision at specific inter- 
vals, but potentially delineating short- and long-term outcomes associated with 
its provision. This can be achieved by using natural language processing (NLP) 
tools like latent Dirichlet allocation or latent semantic analysis in R (https:// 
www.r-project.org/) or Python (python.org) to explore important topics that 
emerge over time; alternatively one could use the Linguistic Inquiry and Word 
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count program (LIWC [20]) to explore psychological constructs that appear in a 
user's tweets (e.g., positive emotion, social words). Several studies reveal that 
when traumatic events like terrorist attacks [21—23], natural disasters [24, 25], 
school shootings [26-28], and other large-scale distressing events (i.e. mass 
communications of impending threats [29]) occur in communities, people 
express their emotions on Twitter. In such contexts, there is ample opportunity to 
explore how community members provide social support and ultimately link the 
receipt of social support to psychological functioning in the weeks and months 
that follow. 

It is also possible to examine the provision of social support in the networked 
communications between Twitter users. Users engage with each other in conversa- 
tion by explicitly tagging members of the conversation, or by tweeting to users 
contained in lists which are user-created and curated for specific purposes. As long 
as the tweets and lists are public, researchers can scrape these data from the plat- 
form. This network of tweets linked between users can give rise to analyses that 
explore whether receiving social support (via tweets from others) functions to bol- 
ster the wellbeing of the users who are targets of support. Using this information in 
conjunction with user-level profile information (e.g., number of followers, baseline 
engagement with others in a particular context) can also illuminate how outcomes 
related to support provision differ by users with a robust social network versus those 
with a meager network. 


Other Social Media Platforms 


Facebook and Twitter are only two social networking sites in a sea of hundreds; 
however, data on most platforms are not accessible to researchers. One exception is 
Reddit, which has garnered some popularity, as a target for social science research. 
Reddit is a public message-board platform that is organized into general and user- 
moderated topical areas (subreddits) in which users post an array of content includ- 
ing text, images and links. As a more traditional topic-related message board 
platform, there are opportunities for analyzing posts for support provision that mir- 
ror much of what has been discussed above. 


Big Personal Data 


Big data are not exclusive to social media platforms. They are generated moment- 
to-moment, every day, by many of the internet-enabled devices individuals use 
regularly. These data are passively logged on a device most of us carry around with 
us: a smartphone. Hundreds of times each day, individuals pick up their smartphone 
and engage with it in many ways—they text and call friends and family; interact 
with strangers and aquaintances on social media applications; map their routes and 
coordinate all the logistics of their lives. By their very nature, smartphone metrics 
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can serve as proxies for social information about people. The number of phone calls 
one makes, the number of times a text or messaging application is opened, the char- 
acteristics of the physical locations one inhabits when opening a mapping applica- 
tion, all tell a story about how much people interact with others every day and can 
characterize the type of environments people tend to be in (crowded public spaces 
versus private homes). Recently, researchers have begun to tap into the information 
embedded in smartphones to understand daily human sociability [30] but they have 
not yet linked this information to the experience or perception of social support in 
peoples' lives. 

How, then, can smartphone data be leveraged to understand anything substantive 
about social support? The answer is muddy at best, but exciting nonetheless. By 
focusing on contextual factors that can be gleaned from passive sensors on our 
smartphones (e.g., GPS locations, recorded conversational elements), we can attempt 
to operationalize certain elements of social support. For example, if a researcher 
wanted to operationalize the question on the MSPSS that states, "I can talk about my 
problems with my friends", they could potentially use smartphone sensors to deter- 
mine proximal locations and linguistic tendencies during social interactions that 
would provide information about whether the individual of interest does indeed talk 
to their friends about problems. This type of approach for operationalizing social 
support is promising for future development, but is limited in its scope by the com- 
plexities of social behavior. Smartphone data cannot stand on their own for the sim- 
ple reason that they are approximations of social behavior and connection. For 
example, just because someone spends a lot of time using social messaging apps 
does not inherently signal that they have a lot of social support. In fact, those digital 
social interactions could potentially be mostly negative. Without explicit access to 
messages in order to analyze the content or tone of the exchange, relying solely on 
logged texting behavior could be misleading. The same is true for leveraging smart- 
phones' Bluetooth capabilities to detect nearby people to gauge how often people 
are around other people. Without more information from the individual about 
whether an exchange occurred, and the nature of that exchange, not much can be 
gleaned from looking at these data without supplementing with self-report data. 

To remedy this, smartphones do, however, offer a powerful way to reveal social 
support processes in real time through ecological momentary assessment (EMA; 
[31], a method that allows researchers to reach people directly through their smart- 
phone. This method involves pinging participants a few times a day (via a notifica- 
tion), at fixed or random intervals, to obtain a snapshot of their emotions, social 
interactions, and other psychological information of interest to the researcher. 
Before smartphones, researchers would have to acquire funds to provide partici- 
pants with handheld devices (e.g., palm pilots) to survey their experiences through- 
out the day. However, the ubiquity of smartphones today has made it easier than 
ever to measure the quantity and quality of daily social interactions by prompting 
individuals to complete brief daily surveys. Imagine, then, coupling EMA data with 
smartphone (or other wearables) sensor data. When paired, these data can allow 
researchers to untangle the complexities of daily life in analytic frames that help us 
understand biopsychosocial processes that unfold in real time. Tapping into the 
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bountiful cornucopia of big personal and social media data represents an exciting 
opportunity for researchers to quantify social support with unparalleled detail and 
accuracy. 


How Does Technology Influence Our Conceptualization 
of Social Support? 


Technology has fundamentally altered the way we interact with others. We are 
increasingly using internet-mediated communication platforms as the primary mode 
of interacting with others. This shift to online social interactions has transformed 
our conceptualization of social support. We understand social support differently 
and it impacts our lives differently as we continue to intertwine our lives with tech- 
nology. In the following sections, we will highlight three areas in which social sup- 
port is evolving due to technology adoption. We will examine how smartphones can 
exert positive and negative influences on face-to-face social interaction. Next, we 
will peer into online communities and explore how social support garnered online is 
influencing the dynamics of health and social support. Finally, we will focus on how 
family parent-child relationships are different in the digital age, and what this means 
for quality of life. 


The Effect of Technology on Interpersonal Communications 


Personal technological devices, such as smartphones, have become a constant com- 
panion in most people's lives and subsequently influence the dynamics of face-to- 
face social interactions. These devices are cognitively distracting, even when not 
actively used [32] and often lead to a state of absent presence [33] in which an 
individual is physically present, but their mind is wandering elsewhere. This distrac- 
tion induced by smartphones is particularly influential during social situations 
because people tend to associate their devices with external social networks [34] 
likely because we use smartphones to call, text, message, share, and communicate 
with our wider social support system. Thus, the mere presence of a smartphone may 
orient someone to think of people outside the context of their face-to-face conversa- 
tion and divert their attention away from a conversational partner. Indeed, qualita- 
tive evidence has demonstrated that smartphones make social networks more salient 
and direct attention away from face-to-face conversations [35]. Interestingly, this 
type of activation of relational schema may take place without a person's awareness 
[36]. The simple presence of a device, consciously or unconsciously, distracts us 
and activates representations of social networks that may exert positive and/or nega- 
tive influences on in-person communication depending on the context of the 
interaction. 
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Several psychological experiments have demonstrated that when an individual 
is distracted from their immediate face-to-face conversation partner(s) due to 
smartphone presence, the quality of that interaction suffers. Researchers found that 
the presence of a phone can have a negative influence on closeness, connection, and 
conversation quality between dyads [37]. In a naturalistic experiment, people in a 
coffee shop who conversed together without smartphones present reported higher 
levels of empathetic concern for their partner than those who had phones present 
[38]. Similarly, groups of friends who ate a meal together without phones present 
reported less distraction and more enjoyment than those who had phones with them 
[39]. Finally, using objective assessments of smiling behavior, researchers found 
that conversation partners who were in the presence of a phone were less likely to 
smile than those who had no phone present [40]. The detrimental effects of having 
a smartphone present during potentially positive social interactions have implica- 
tions for social support and quality of life. By lowering the quality of the interac- 
tion, phones may interfere with the formation of new relationships and disrupt the 
maintenance of existing relationships. In this way, smartphones themselves, even 
when not used, may have a negative influence on social support in our mod- 
ern world. 

However, the presence of a smartphone may not always lead to negative conse- 
quences. In undesirable social situations, such as stressful or isolating interactions, 
smartphones may actually provide benefits. People can rely on smartphones as an 
avoidant coping mechanism, as demonstrated by an experiment that showed how 
the presence of a smartphone can lower an individual's initial reaction to social 
stress [41]. Another experiment demonstrated that the mere presence of a phone can 
aid in recovery from a socially stressful situation. Individuals who had their phones 
with them, but were restricted from using the devices, exhibited sharper declines in 
physiological stress after they were exposed to a stressful social exclusion paradigm 
compared to those who had no phone or used their phone [42]. The representational 
image of our smartphone may increase feelings of perceived support and provide a 
reminder of the social resources available to cope with a stressor at hand. As dis- 
cussed earlier, perceived social support is a key predictor of health because of its 
ability to help us handle stress. So if smartphones symbolize perceived social sup- 
port, they may be relied upon to help us overcome stressful situations. In this way, 
phones can be health-protective by serving as something akin to a digital security 
blanket that offers comfort in uncomfortable circumstances. 

The distracting pull of smartphones on our attentional awareness is responsible 
for both the positive and negative effects mentioned above. Specifically, distraction 
caused by the symbolic connections offered by phones can shift attention away 
from negative environmental stressors and provide a sense of security. On the other 
hand, the salience of social networks represented by a phone can pull attention away 
from a potentially positive interaction and lower the quality of that conversation. In 
both cases, the key element that allows the mere presence of a phone to exert these 
positive or negative influences is the symbolic social connections represented by a 
smartphone. The way we conceptualize social support has altered significantly due 
to the widespread adoption of smartphones. The digital threads that connect us to 
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our wider social support system are in our pockets at all times, and carrying that 
symbolic network of friends and families with us can be positive or negative depend- 
ing on the circumstances of our face-to-face interactions. 


Online Social Support 


Online communities, such as the aforementioned Facebook groups, provide abun- 
dant opportunities for users to give and receive social support. These online interac- 
tions—and the support and strain users derive from them—is similar but distinct 
from that exchanged offline. Much like support exchanged in offline groups like 
Alcoholics Anonymous and hobby clubs, giving and receiving online social support 
can be a source of validation contributing to an improved quality of life. Online sup- 
port can take the form of advice written in a reply to a weight loss forum post, a 
“like” on an Instagram photograph, or banter amongst high school friends in a 
groupchat. Likewise, norms around social support exchange differ across online 
locales; for example, social networking sites (SNSs) like Facebook can be con- 
trasted with anonymous online support groups (OSGs) composed of strangers fac- 
ing a shared challenge, like the subreddits/depression. Despite the challenge of 
defining online social support, seeking social support it is one of the foremost rea- 
sons that individuals choose to participate in online communities. 

The unique dynamics of Internet-mediated social interactions shape the ways 
that users exchange support. The relatively low stakes of online interactions mini- 
mize typical in-person impediments to conversation and relationship formation 
(e.g., shame, stigma, appearance, and physical inability) [43]. Online interactions 
enable users to express their ‘true selves’ more than they would in person [44—46], 
creating the potential for “hyperpersonal interactions” [47] featuring high openness 
and liking between parties. Even relationships that exist entirely online can be 
meaningful sources of social support [48]. However, loosened social norms online 
also facilitate the misinformation and bullying for which online communities are 
notorious [49]. 

Numerous positive impacts of OSGs have been identified, although their impacts 
on “hard” health outcomes need to be more rigorously investigated [50]. Perceived 
social support from an OSG is mediated by identification with the community and 
interpersonal bonds with other members [51]. Similarly, identification with other 
forum members is a key moderator of the link between positive psychosocial out- 
comes and participation in online discussion forums [52]. Research has also demon- 
strated that Facebook-based social support generally improves physical and mental 
health, and reduces symptomatology related to mental illness [53]. 

Online sources of social support are particularly crucial for people who struggle 
to find support offline [54, 55]. One study found that typical inequities in support 
availability related to race and age are minimized among those that have access to 
the internet and SNSs [56]. Using massive social networking sites like Twitter, peo- 
ple facing rare or understudied health issues can rapidly connect to exchange 
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first-hand experiences and expert insights. For example, after he was diagnosed 
with COVID-19 early in the pandemic, the digital health speaker Maneesh Juneja 
used his twitter to publicly share frequent updates on his recovery and recommend 
digital resources to others. Another unique benefit of OSGs is that they provide 
forms of support that are specific to the needs of their users. For example, members 
of Mood Disorder communities offer one another primarily emotional support, 
while Compulsive Disorder community members tend to exchange instrumental 
support in the form of tips for dealing with symptoms [57]. 

Providing support to others, while often fulfilling, can also be quite taxing. Some 
SNS users feel overwhelmed by the inundation of support requests they encounter 
from other members of their online communities (i.e. a Facebook status asking for 
help moving or a Twitter post venting about a difficult breakup). This feeling of 
"social overload" is particularly common for users who feel they are obliged to 
respond to SNS support requests, as well as those who have a greater number of 
online-only friends, as compared to friends with whom they have offline relation- 
ships as well [58]. Similarly, mental-health OSG users often complain of "endless 
grief loops" from encountering an excess of disheartening stories from other users 
[59]. These concerns, as well as the prevalence of trolling, bullying, and misinfor- 
mation under anonymity, present substantial downsides to participation in online 
spaces for some [60]. 

Online communities' scale and accessibility enable users to bypass common bar- 
riers to giving and receiving social support. These spaces hold particular appeal for 
people who lack the in-person networks to openly discuss the topics they care about. 
At their best, online communities offer empowerment through genuine human con- 
nection. Yet, because people interact online in such diverse ways, more research is 
necessary to fully understand how online social support contributes to users' broader 
social lives. 


Parent-Child Relationships 


Frank Lloyd Wright's well-known quote, “the hearth is the psychological center of 
the home" conjures images of quiet reflection, children cuddling with parents, sto- 
rytelling, and bonding with loved ones. Today's switch to an electronic epicenter of 
the home may seem abrupt but it has been happening incrementally in many cul- 
tures across the globe for decades. Indeed, the popularization of television in the 
1950s marked the beginning of the screen as a replacement for the hearth, followed 
by the rise of the desktop computer in the 1980s, the mobile phone in the 1990s, 
smartphones in the 2010s, and more recently, social media and online gaming 
fiercely vies for attention at the family dinner table. Yet, as tempting as it may be to 
romanticize times passed, it is likely that the drive for social connection has not 
changed from one generation to the next, rather, it is the vehicle of connection that 
has radically transformed. Research on the impact of this technological shift on 
family relationships is quite mixed with some studies suggesting that it is 
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detrimental to the family bond while other studies show that technology, in its vari- 
ous forms, promotes healthy connection. Benefits and detriments alike, most fami- 
lies have invited this virtual guest into the home and technology has established a 
firm seat at the table. 

An electronic third-party is now in near constant attendance, at times enhancing 
interactions with close loved ones and at other times, detracting from them. Many 
children are raised by parents tied to mobile devices with popular media and large 
portions of society expressing concerns regarding the repercussions on children's 
well-being due to distracted parenting. These concerns are not completely 
unfounded. Some research has shown that screen time is detrimental to the parent- 
child relationship. Specifically, distracted parenting due to mobile device use is 
linked to more child behavior problems, increased risky-behaviors in children, 
reduced parent-child interaction, and reduced parental sensitivity [61]. Further, chil- 
dren notice parents' use of mobile devices and report feeling excluded and emotion- 
ally dissatisfied during these occurrences [62]. However, this evidence should not 
be over-simplified and is only one piece of a much larger body of literature. Research 
also shows that a brief distraction may help parents to recharge and re-engage with 
their children more effectively. For example, one study found that following approx- 
imately 15-min of focused mobile device use; parents often initiated exuberant and 
joyful play with their children [63]. In this sample, parents first ensured that chil- 
dren were engaged in safe play prior to using their mobile phones. It was only then 
that most parents began smartphone use. After this well-placed distraction, parents 
re-engaged with more enthusiasm and interest than they displayed pre-distraction. 
The researchers described an ebb and flow between engagement and disengagement 
and suggest that the cycle between interactions may provide relational benefits. 
While mobile phones may foster a type of disengaging recharge for parents, other 
technologies appear to facilitate parent-child engagement. 

The majority of American homes have a television as the focal point of the fam- 
ily room and the TV is one screen designed to host a group experience. Research has 
found that parental co-use of technology may have advantages [64], for example, 
co-viewing of television is linked with gains in preschoolers comprehension [65], 
attenuated fear and aggression among school-aged children [66], direct positive 
effects on language development in low-income immigrant families [67] and may 
help with young children's verbal development in well-designed programming [68, 
69]. Beyond television, there is also evidence supporting the benefits of parent-child 
co-use of mobile phones, gaming systems and computers. Research shows that teen- 
agers benefit from parental help with computers and this may promote self-efficacy 
and technological expertise [70—72]. Further, parents that played video games with 
teenage daughters had daughters that reported higher parental connection, fewer 
internalizing problems, and increased prosocial behaviors than those that did 
not [73]. 

Advantages and disadvantages alike, technology is a firmly entrenched presence 
in most homes with the number of smartphone subscriptions surpassing the world 
population [74]. Even still, parents who use technology with children present bear 
the brunt of heavy criticism and regular shaming from the popular media. But 
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parental distraction is not a new phenomena and is certainly not unique to the current 
generation. In times past, parental heads might have been buried in a newspaper or 
absorbed in home projects, hobbies or social clubs. The fact that the new distraction 
is digital does not make it inherently harmful and undivided attention from parents 
to children is not necessarily beneficial for either party. The challenge is found in the 
ability to discern how to calibrate and modify technological use, so that it facilitates 
familial well-being. Technology can be used as a vessel that helps to hold relation- 
ships in the same space or it can be used to divide. It is neither good nor bad until the 
user gives it its purpose. One could argue that parents who use mobile devices to play 
games with their children, photograph memorable moments, and connect with other 
parents better serve their children. Further, the co-use of technology gives parents an 
opportunity to influence how children navigate their way through virtual networks. 
Indeed, school-aged children that co-use the internet with parents are more likely to 
seek out educational websites when compared to children who co-used the internet 
less [75] and respected organizations are adjusting recommendations based on 
emerging evidence. In 2013, the American Academy of Pediatrics changed its rec- 
ommendation from strictly limiting media for young children to encouraging parents 
to co-use media with their children [76]. Taken together, the collection of informa- 
tion we have discussed suggests that the influence of the digital presence on familial 
relationships fluxuates based on contextual cues and is more likely a reflection of the 
psychology of the user rather than the influence of the tool itself. 


Discussion 


The integration of technology into our daily lives has broadened the scope of 
interaction with our social networks and expanded the ways in which we can 
assess the influence of social support on health and quality of life. The abundant 
and detailed data produced by our online activity and technology-infused life- 
styles represents a fertile ground for exploration into the intricacies of social 
interactions. Since social support is such a critical aspect of quality of life, it is 
imperative that we continue to develop innovative methods for assessing social 
interactions. A focus on Quality of Life Technologies (QoLT), the software and 
hardware that allow us to assess and monitor well-being [77], may allow us to 
take this next step forward in capturing and disseminating social information 
about our lives. By leveraging these QoLT, such as the social data recorded on our 
smartphones, we can hope to better understand social interactions and draw upon 
the multitude of opportunities for improvement of well-being. The omnipresence 
in social circles coupled with the hardware and software capabilities of QoLT 
allow for an unprecedented level of insight into the dynamics of social well-being 
that can be drawn upon to assess and improve social support. Traditionally, 
researchers would solely rely on self-report measures to assess social support and 
its related constructs. And while this data is certainly valuable and important, the 
objective and unbiased information gathered from technology-enabled methods 
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provides an unprecedented level of detail and insight that uncovers the dynamic 
and complex nature of how social support unfolds. In tandem with social implica- 
tions, mobile technologies offer us the ability to track and measure health indices 
such as sleep, exercise, nutrition, and cardiovascular function. Together, these 
tools can be employed as electronic observers providing insight into the various 
ways social behaviors (on and off-line) may impact important health behaviors 
and outcomes. 

By drawing on a multitude of technological resources, researchers are able to 
leverage big data to examine the complex ways in which social support transpires 
in the modern age. By focusing on social media activity of groups and individuals 
via platforms like Facebook and Twitter, researchers can begin to understand how 
social interactions manifest online. This data can be linked to important individual 
psychosocial outcomes or health-relevant group concepts that may inform how 
online social support influences health. Furthermore, the plethora of technological 
devices that pervade our daily lives can be harnessed to provide big personal data 
that informs our understanding of social support. Researchers can glean informa- 
tion from smartphone behavior or wearable devices to passively track socially- 
relevant factors that occur in real-time in the real-world. We can also utilize 
techniques such as ecological momentary assessment to gather in-vivo measure- 
ments of daily social and well-being variables. This information may be used by 
researchers to update theories and ideas about the biopsychosocial effects of social 
support, and may also be relied upon to inform health practitioner recommenda- 
tions or interventions. Additionally, individuals can analyze their own digital social 
metrics to better quantify their social wellness and recognize areas for potential 
improvement or change. By engaging in this way in the Quantified Self movement, 
individuals may be able to augment their quality of life by creating data-driven 
goals for behavior change. Taken together, the advent of internet-enabled techno- 
logical devices has opened a never-before-seen window into the intricacies of indi- 
viduals social lives and health behaviors that allow us to capture a wide array of 
biopsychosocial information that can be leveraged to better our understanding of 
these quality of life indicators. 

When considering the meaning of this technology derived social data, it is criti- 
cal that researchers and practitioners also keep in mind the ways in which technol- 
ogy has altered our understanding about the conceptualization, meaning, and 
influence of social support in the digital age. The intrusion of technology into our 
social interactions has innumerable positive and negative effects on the quantity, 
quality and function of social support. Social support is increasingly taking place in 
the realm of internet-mediated communications, and this transition to virtual com- 
munication alters the applicability of and relevance of the traditional ways in which 
we understand the impact social support on well-being. In this chapter, we focused 
on three areas in which technology has changed our conceptualization of social sup- 
port, namely in regard to face-to-face conversations, social support groups, and 
family interactions. 

As smartphones have come to symbolize social networks, due to their use as 
social communication devices, the presence and/or use of these devices has altered 
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the dynamics of face-to-face conversations. The symbolic social support repre- 
sented by smartphones distracts us from in-person interactions by unconsciously or 
consciously drawing our attention away from the present situation. This absent 
presence often decreases the quality of our face-to-face interactions, especially 
when that interaction is potentially positive. On the other hand, that same social 
distraction can be beneficial in aversive situations when the symbolic connections 
provided by our smartphone provide a temporary digital security blanket crutch on 
which we can rely on to buffer the stress experienced in the undesirable 
circumstance. 

Similarly, our interactions in online communities introduce an extra layer of 
complexity into how group-level social support influences well-being. Online 
communities, ranging from massive social media platforms to niche hobby 
forums, are an evolving and influential social phenomenon. These spaces enable 
interactions distinct from real-world groups, presenting unique opportunities and 
challenges. For some, online communities can be a gateway into progressively 
darker mindsets, yet for others they are a lifeline, offering hope in the form of 
genuine human interaction and social support. Our online activity deeply influ- 
ences our lives and society on a global scale, and as such deserves careful 
attention. 

Family dynamics are also undoubtedly shifting due to the widespread adoption 
of smartphones and the internet. The digital presence in many households can often 
drive a virtual wedge between family members and lead to developmental or rela- 
tional problems when devices are overused. Further, whether technology is used to 
promote the quality of family relationships or not, any overuse of screens could 
promote sedentary behavior ultimately impacting health and well-being. On the 
other hand, technology can be used to bring families together, foster communal 
experiences, and increase familial engagement. As technology cements its seat at 
the dinner table, it will be critical for parents (and children) to be aware of the posi- 
tive and/or negative ways in which their digital behavior influences family well- 
being. Families will need to learn to use technology in moderation and operate this 
electronic tool with the purpose of fostering positive health behaviors. In sum, a 
verdict cannot be assigned as to whether technology is “good” or “bad” for families. 
Like any tool, it can be abused and misused or in can become an instrument boost- 
ing the quality of life for many. 


Future Directions 


There are innumerable possibilities for how technology will influence our assess- 
ment and conceptualization of social support in the coming years. Technological 
change is occurring at a blistering pace and it is nearly impossible to predict exactly 
how it will impact our society and individual well-being. We remain optimistic that 
the breadth and depth of our knowledge and understanding of social support will 
continue to expand. 
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Technology-enabled methods and techniques for objectively assessing social 
support will likely flourish in the coming years and provide an unprecedented level 
of detail about our social lives. Researchers can leverage passive sensors to capture 
behavioral observations like facial expression, tone of voice, or body posture along 
with location or usage information to further understand the nature of social interac- 
tions. Of course, there are privacy issues that will need to be considered with such 
approaches. But individuals who are willing to provide this access to their informa- 
tion will help researchers and practitioners capture more fine-grained details about 
the nature of social support. The amount of data accumulated through online activ- 
ity, smartphone behavior, wearables, and yet-to-be invented technological devices 
will paint a complex picture of social life, and the social interactions therein. These 
data will be leveraged by researchers and practitioners in order to find ways of 
improving our social relationships and long-term wellbeing through rigorous 
research methodologies, data mining, and targeted interventions. There are so many 
different streams of data, some on the individual level and others on the group level, 
that will accrue information about social happenings across the globe. As wearable 
technology and smart devices become ever more integrated into our lives, the 
moment-to-moment details of our existence will inevitably leave a digital trace. If 
researchers can find a way to funnel this information together to develop compre- 
hensive social profiles of individuals or groups, then that information could be used 
to provide unprecedented insight into the manifestation of social support. 

As seen in the sections above about the influence of technology on social sup- 
port, the intersection of these devices with our traditionally understood reality can 
produce mixed results. Depending on contextual factors, individual characteristics, 
motivational reasons, and conscious or unconscious behaviors, technology can 
either wreak havoc on the quality of social relationships or supplement our connec- 
tions and enrich social activity. One factor that seems to be overwhelmingly positive 
in regard to the use of technology and well-being is when technology is used com- 
munally (i.e., watching a video or playing a game together) rather than solitarily. 
While this kind of shared experience does not necessarily promote conversation in 
the moment (depending on the viewing choice) it can promote touch, warmth, 
shared suspense, and excitement when users choose to utilize it this way. 

One possible trend of interest to researchers and practitioners that could continue 
to develop is the displacement of the functions of social support with internet- 
mediated support. Traditionally, we think of social support providing benefits by 
offering emotional, informational, or instrumental resources that aid an individual 
in times of need. For most of human existence, if someone had a stressful problem 
to deal with they would most likely rely on significant others for advice, comfort or 
resources. But what if technology itself, rather than another human being, provided 
this support? Indeed, many people already rely on technology to provide support in 
times of need. Googling a question about a novel health concern may provide more 
useful information than asking your neighbor, interacting with a caring avatar in a 
video game (or a caretaker robot) may be able to give someone a sense of comfort 
and security that busy friends cannot provide, soliciting strangers to crowdfund for 
a personal cause may be more effective than asking a family member for a loan. In 
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these instances, direct human contact has been taken out of the equation. As people 
become more reliant on digital means of communication, individuals may reap the 
benefits of instrumental, emotional, and tangible support through Internet use, 
rather than through face-to-face social relationships. By considering this possibility, 
we can see how technology has opened up the doors to a whole new network of pos- 
sibilities for support sources that could conceivably replace the traditionally under- 
stood social support. 

Yet, we do not believe that society will evolve (or devolve) into a place where 
face-to-face human connection has been subsumed by virtual environments. It is a 
frightening thought to imagine a future where individuals do nothing more than sit 
alone and stare endlessly into a glowing screen that seamlessly provides for all their 
needs and desires. I believe that even as we climb ever higher up the ladder of tech- 
nological innovations, humankind will always be at the core of it all. People have an 
innate desire to seek social connection [78], and that desire will ensure that we 
never stray too far from reliance on our place-based social networks. Even as we 
rely more heavily on non-human sources of support (i.e., Internet), it is important to 
remember that people are the ones who create the content of the Internet. While 
attempting to reduce your stress or receive support by asking Google a question 
might seem like an entirely non-social activity, a human-being was the one who 
actually wrote the answer that you seek. So instead of thinking about technology 
replacing social relationships, it may be more appropriate to view technology as a 
medium that can supplement and broaden the ways in which social connections play 
out. We can use the Internet to strengthen our current social bonds, to expand our 
social networks, and most importantly to draw on the worldwide human experience 
to provide the support that will allow us to flourish. 


Conclusions 


Technology-enabled methods have allowed for more accurate and detailed quantifi- 
cation of social support. By analyzing the digital traces of individual and group 
behavior, we are able to better understand the complex dynamics of social interac- 
tions and its influence on well-being. Continued advancements in technological 
approaches will likely further enhance our comprehension of social support and 
equip individuals, researchers, and practitioners with the necessary knowledge and 
tools to improve quality of life. 
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Chapter 16 A 
Sexual Function and Quality of Life: inim 
Assessing Existing Tools 

and Considerations for New Technologies 


Diana Barger 


Introduction 


Sex or sexual activity is a core human function. It refers to the way in which people 
experience or express their sexuality. It encompasses activities done alone (e.g. 
masturbation) or with another person (intercourse, oral sex, etc.) or people (group 
sex) [1]. Sexual activity is considered to be an important part of social functioning 
(also referred to as social health), which is distinct from mental and physical health 
and seen as an important dimension of quality of life (QoL). Echoing its definition 
of health, the World Health Organization's (WHO) has defined sexual health as a 
"state of physical, emotional, mental and social well-being in relation to sexuality; 
it is not merely the absence of disease, dysfunction or infirmity" [2]. The WHO has 
acknowledged its importance by including an item on sexual activity in its assess- 
ment of QoL. The item “how satisfied are you with your sex life?" seeks to capture 
the desire for, expression of, opportunity for and fulfillment from sex. 

These definitions differ somewhat from the assessment of sexual function, which 
has been defined as “how the body reacts in the different stages of the sexual 
response cycle", first described by pioneering researchers Masters and Johnson in 
the late 1960s [3]. The sexual response cycle was defined as linear stages: (1) excite- 
ment, (2) plateau, (3) orgasm, and (4) resolution, all of which are experienced by 
both men and women [3]. This model of sexual response has since been criticized 
for its overt focus on the physiological rather than psychological and social compo- 
nents of human sexual response. Subsequent models of human sexual response have 
aimed to correct these shortcomings. Kaplan incorporated desire into the model [4]. 
Whipple and Brash-McGreer and later Basson proposed alternative, more complex, 
models of sexual response in women that were circular rather than linear [5, 6]. 
Basson's model specifically recognizes that orgasm may contribute to, but is not 
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necessary for, satisfaction, and emphasizes that relationship factors can affect one’s 
willingness and ability to participate in sex. 

As our understanding of sexual response has evolved, so have accepted diagnos- 
tic criteria for sexual dysfunctions (SD). The Diagnostic and Statistical Manual of 
Mental Disorders (DSM)—V, published in May 2013, sought to incorporate newer 
models of sexual response and to correct, expand, and clarify criteria for diagnoses 
[7]. Definitions of male sexual dysfunction (MSD) according to the DSM-V include 
erectile disorder, male hypoactive sexual desire disorder, premature (early) ejacula- 
tion, and delayed ejaculation (previously termed male orgasmic disorder). MSD 
involving pain was removed from the DSM-V. Definitions of female sexual dys- 
function (FSD) in the DSM-IV were collapsed in the DSM-V. FSD includes female 
sexual interest/arousal disorder (previously female hypoactive desire disorder and 
female arousal disorder), female orgasmic disorder, and genito-pelvic pain/penetra- 
tion disorder (previously dyspareunia and vaginismus). The DSM-V also required 
that frequency (dysfunction present 75—100% of the time), duration (a minimum of 
6 months), and distress be investigated in order to make a diagnosis. 

Although epidemiological studies are scarce, SD appear to be prevalent in the 
general population and varies by sex and age. In a large United States-based survey, 
conducted in 1992, in adults age 18—59; 43% of women and 31% of men reported 
experiencing SD over the past 12 months based on the DSM-IV criteria [8]. 
Disorders of desire/interest, arousal, orgasm and pain appear to be more common in 
women than in men. The most common complaint in men is premature ejaculation 
[8]. The causes of SD can be either physical such as common chronic conditions 
and/or their treatment (e.g., diabetes, heart and vascular disease, hormonal imbal- 
ance, alcoholism, etc.) or psychological (e.g., stress, depression). 

The high prevalence of SD and its impact on individuals’ QoL prompted the 
development of pharmacological interventions for addressing these disorders (e.g. 
erectile dysfunction). It was therefore necessary to develop valid and reliable meth- 
ods for both diagnosing SD and assessing the impact of experimental interventions 
on sexual function and QoL. There are a number of direct physiological measures 
of sexual function used in men and women. For example, erectile function can be 
diagnosed using a Nocturnal Penile Tumescence (NPT) device [9], Intracavernous 
Injection of Prostaglandin E1 [10], or the penile/brachial index [11], Doppler stud- 
ies [12] and sacral evoked potentials. In women, genital blood peak systolic velocity 
[13], vaginal pH, intravaginal compliance, and genital vibratory perception thresh- 
olds are used as direct measures of female sexual function. These direct measures 
appear to be correlated with indirect measures, specifically hormonal changes 
within the body, like estrogen, luteinizing hormone, testosterone and prolactin. 
However, they are useful insofar as for a diagnosis of sexual (dys)function rather 
than for the assessment of sexual QoL. In addition to direct and indirect objective 
measures, sexual function and more importantly sexual QoL can also be assessed 
via standardized inventories or self-administered questionnaires. These vary in their 
scope (comprehensiveness), measurement properties (evidence of validity, reliabil- 
ity and responsiveness), applicability (e.g., use in women, LGBTI populations) and 
context (e.g., research versus clinical practice). We hypothesized that although there 


16 Sexual Function and Quality of Life: Assessing Existing Tools and Considerations... 397 


are numerous instruments, covering sexual function and QoL, not all were suffi- 
ciently comprehensive nor were they applicable to all populations. 

Furthermore, the use of new technologies have enabled not only the collection of 
self-reported measures of sexual function and sexual activity in relation to sexual 
and reproductive health, but may facilitate the collection of indirect measures of 
sexual activity via applications or personal, miniaturized mobile and wearable 
devices. The private sector has responded to people’s interest in better understand- 
ing not only their reproductive health, for instance with the use of menstruation and/ 
or fertility awareness applications or devices [14], but also, their sexual activity. 
Mobile phone applications for the assessment of sexual activity run the gamut and 
allow users to quantify and qualify sexual activity. These applications differ sub- 
stantially from those aimed at individuals or couples trying to conceive or avoid 
unwanted pregnancy. Many of these applications still rely on users completing an 
assessment, reporting and often qualifying their sexual activity, whereas others use 
the smartphone’s sensors or those of personal devices (Smart watches or rings or sex 
toys) to log sexual activity. These applications, wearables and devices might offer 
new opportunities for research and clinical practice. We discuss these opportunities 
in relation to commonly used methods, e.g., self-administered questionnaires, in 
this chapter. 


Aims 


We conducted a review of literature reviews of self-administered assessments of 
sexual function and QoL to identify viable instruments for the assessment of sexual 
function in the context of research and clinical practice. We then evaluated their 
breadth (dimensions assessed), summarized evidence of their psychometric proper- 
ties and applications (e.g., clinical research/screening) and considered these within 
the context of emerging technologies. 


Methods 


We conducted a meta-review of literature reviews focusing on instruments assessing 
sexual function or sexual QoL. We identified relevant literature reviews from two 
sources, the COnsensus-based Standards for the selection of health Measurement 
INstruments (COSMIN) database! and the academic bibliographic databases, spe- 
cifically Pubmed, PsycInfo and Scopus, using a search algorithm. The former is 
maintained by COSMIN, an initiative that aims to improve the choice of 
measurement instruments for research and clinical practice [15]. The following 
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algorithm was used (“sexual satisfaction" OR “sexual quality of life" OR “sexual 
well-being" OR “sexual function" OR “sexual dysfunction") AND (“questionnaire” 
OR “patient-reported outcome" OR “measure” OR *"scale") AND (“review”). The 
choice of key words, ranging from “quality of sexual life" to “sexual dysfunction", 
reflects the lack of consensus regarding definitions of sexual QoL previously evoked. 
Moreover, in the first search attempts, particularly when using biomedical search 
engines, it was clear that the terms “sexual dysfunction" and “function” were com- 
monly used to describe the instruments of interest. Including these terms ensured 
that instruments of interest, that we might have otherwise missed, were identified. 
We restricted the search fields to titles and abstracts and used search filters to refine 
the result. We restricted our search to "reviews" or "systematic reviews", available 
in English. No restrictions were imposed based on the date of publication and sys- 
tematic reviews as well as literature reviews were considered. We choose to include 
non-systematic reviews as well as systematic reviews to ensure that no relevant 
instruments were omitted. Relevant references were selected based on a review of 
their title, followed by a review of the abstract by two independent reviewers. In the 
event of disagreement, the third reviewer provided an opinion. Full texts were then 
retrieved and screened for inclusion. The results from each database were imported 
into the Zotero,” free open-source reference management software. Zotero was used 
to identify and eliminate duplicate references. 

Included literature reviews of instruments served to aid in creation of a database 
of instruments used to assess aspects of sexual function/dysfunction and/or sexual 
QoL of life in adults. Duplicate instruments were deleted. We then further selected 
instruments based on the following criteria. We first considered the date of develop- 
ment, preferring to include instruments developed after 1970 to account for changes 
in sexual and social norms (e.g., women's liberation and gay rights). We opted to 
focus this review on instruments, which were generic in nature, meaning that they 
had not been developed for the evaluation of sexual function/QoL in a specific 
patient population. The focus of this review was on self-reported measures and 
therefore physician or researcher administered tools were excluded from this review. 
We only included instruments which were associated with a peer-reviewed publica- 
tion. This was re-assessed in case it had not been a requirement for inclusion in the 
individual reviews. We were interested in the broad assessment of sexual QoL and 
not merely the physical aspects of sexual function. We chose to exclude instru- 
ments, which were too narrowly focused to the physiological dimensions of sexual 
function (e.g., arousal, performance, orgasm). We did not distinguish between 
instruments intended for research and those intended for clinical practice at the 
selection stage; yet we did note whether the selected instrument was intended for 
research, clinical practice or both when evaluating the selected instruments. 

The evaluation of the selected instruments covers both their content and the mea- 
surement properties. We wanted to understand whether the existing instruments 
were sufficiently comprehensive (number and extent of the areas covered), 
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reflecting the complexity of the quality of sexual life. This constitutes an assessment 
of their content validity, as measurement instruments must adequately reflect con- 
struct of interest, in this case sexual QoL. We coded the facets that the instruments 
purportedly covered according to two broad categories: sexual function and sexual 
QoL. Items that covered desire and the three physiological stages of sexual response: 
arousal, plateau (maintaining arousal) and orgasm, and those evaluating the pres- 
ence of pain were considered to be part of the evaluation of physiologic sexual func- 
tion. Additional items assessing satisfaction, relational-emotional aspect of sex, 
self-esteem or body image, and finally, the importance of distress for the individual, 
were considered to pertain to the subjective, often psychological, dimensions of 
sexual function, which together with the physiologic dimensions encompass sexual 
QoL. Furthermore, the anchoring of sexual activity and function with regard to its 
impact of QoL is important, as this is part of the current criteria for dysfunction. For 
the evaluation of the psychometric properties of the included instruments, we used 
the COSMIN taxonomy of measurement properties, based on an international con- 
sensus about the relevant properties to be evaluated for any measurement instru- 
ment, regardless of its application. We summarized evidence of the instruments’ 
reliability, validity and responsiveness (10). 


Results 


Selection of Reviews of Instruments 


Our search strategy identified 613 references, 133 of which were duplicates, result- 
ing in 480 references. Based on a review of their titles, 411 references were further 
excluded, as they did not pertain to sexual QoL measures. Sixty-nine abstracts were 
reviewed independently by two reviewers. There was disagreement regarding the 
inclusion of 15/69 (22%) references and these were resolved by a third reviewer, 
ultimately resulting in the exclusion of an additional 58 references, 18 of which 
were excluded as they pertained to instruments used in specific patient populations. 
Eleven full texts were retrieved and reviewed and two additional references were 
excluded because they were not reviews of instruments for the evaluation of sexual 
QoL. The final number of “generic” reviews included was therefore nine [16—24] 
(Fig. 16.1). The characteristics of these reviews, i.e., date of publication, objectives, 
populations studied, and number of instruments evaluated, are reported in Table 16.1. 
Their inclusion criteria, search engines and keywords used are reported in Table 16.2. 
The reviews were published between 2002 and 2018, five of them were systematic 
reviews and the majority (7/9) targeted the general population. One review was 
focused on women and another on homosexual men. Most reviews included only 
validated instruments. In terms of the quality of the reviews, we noted that certain 
methodological standards for reviews of instruments were respected: the majority 
of reviews clearly stated their objective and their predefined inclusion criteria. 
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Records identified through Additional records identified 
database searching through other sources 
(n = 603) (n = 10) 


Identification 


Records after duplicates removed 
(n = 480) 


Records screened on title 
(n = 480) 


Records excluded 
(n = 411) 
Records excluded 
(n = 58) 


Full-text articles assessed Full-text articles 
for eligibility excluded 
(n = 11) (n = 2) 


Records screened on abstract 
(n = 69) 


Eligibility 


Studies included in 
qualitative synthesis 
(n= 9) 


Fig. 16.1 Flow diagram illustrating the selection of reviews for inclusion 


However, nearly all of the reviews did not explicitly state how they selected refer- 
ences, or whether they complied with a standardized method of instrument evalua- 
tion. Reviews tended to focus on instrument validation studies and the instruments 
themselves. They also were interested in the instruments' psychometric properties. 
The definition of sexual QoL appears somewhat nebulous. The number of instru- 
ments included varied significantly between reviews. For the sake of accuracy, the 
information summarized was extracted verbatim from the reviews themselves. 


Selection of Instruments for Consideration 


A total of 110 instruments were included from the nine reviews retained after the 
elimination of duplicate records (instruments). Subsequently, six instruments were 
excluded because of the date of their development (prior to 1970), four were 
excluded because they did not aim to measure sexual QoL, and two instruments 
were excluded because they seemed to lack a peer-reviewed validation study. Of the 
remaining 98 instruments, we retained 34 that were generic measures of sexual QoL 
and excluded the remainder (N = 64) that were instruments designed for use in a 
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Review 1 Review 2 Review 3 Review 4 Review 5 Review 6 Review 7 Review 8 Review 9 
N=25 N=5 N=56 N=30 N=10 N=26 N=7 N=17 N=10 


Instruments retained after duplicates removed 


76 duplicate records 


Instruments excluded: 

* Dates prior to 1970 (N = 6) 
Not designed to measure 
sexual quality of life (N =4) 
Instruments without published 
validation studies. (N=2) 

Instruments included for * Disease-specific instruments 
consideration (N-64) 
(n2 31) * Instrument focused exclusively 
on sexual function (N=3) 


Fig. 16.2 Instrument selected from reviews for consideration 


specific patient population. Finally, three of these were excluded because their focus 
was too narrow, exclusively on sexual function. Thirty-one instruments were ulti- 
mately selected for detailed review (Fig. 16.2). 


Characteristics of Included Instruments 


The characteristics of the instruments included in our study are presented in 
Table 16.3. The dates of instrument development and/or validation range from 1973 
to 2018. Most of the instruments were developed for use in research and clinical 
practice (77% [24/31]) and approximately half (51%, [16/31]) targeted both males 
and females or heterosexual partners. The evaluation of the instruments with respect 
to the dimensions they consider in measuring sexual QoL is reported in Table 16.4. 
This evaluation allows us to judge the adequacy of an instrument in capturing the 
complexity of this construct. As expected, instruments often focused on physiologi- 
cal sexual function or phases of the sexual response cycle. On the other hand, psy- 
chosocial questions regarding relationships/emotional aspects of sex, body image 
and self-esteem cultural and gender expectations were only rarely taken into account 
in the assessment of sexual QoL. The most commonly assessed domains were 
arousal (N = 19), desire (N = 17), satisfaction and pleasure (N = 17), and frequency 
(N = 14). The most comprehensive instrument appears to be the Derogatis Sexual 
Functioning Inventory (DSFI), covering nine dimensions of sexual QoL. The DSFI 
was one of the few instruments to assess distress, relationship issues, fantasies and 
cultural and gender roles. The importance of sexual activity for the individual was 
only assessed in six of the 31 instruments reviewed. Of note, we found that the 
majority were not developed with the involvement of lay individuals or those coping 
with SD. 
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Table 16.3 Content of the included instruments (N = 31) 


PHYSIOLOGICAL DIMENSIONS SUBJECTIVE/PSYCHOLOGICAL DIMENSIONS 
Phases of sexual function 
p Relationship. 
Behavior, | 'mPortance Satisfaction; Anxiety  Bodyimage — " pee 
Instruments-Yearofvalidationstudy | Desire Arousal Performance Orgasm | Pain — & of sexual Pleasure, - - Fantasy 
frequency Feelings and gender roles 
activity Quality Distress Self esteem 
emotions 
1. Sexual Orientation Methods and N Y N N N Y N N N N N N N 
Anxiety (SOMA) ~ 1973 /2018 [25] 
2, Sexual Interaction Inventory Scale N Y N N N 8 ii Y N Y Y N N 
(partner) (S) 1978 [26] 
3. Questionnaire Measure of Sexual 
Interest (QMS!) - 1974 [27] 
4, Derogatis Sexual Functioning Inventory Y Y N N N Y N Y Y Y sf Y Y 
(DSFI) - 1979 [28] 
5. Female Sexual Arousability Index (SAI) Y Y Y N N $ N N N N N N N 
1989 [29] 
6. Index of Sexual Satisfaction (ISS) — Y Y N N N Y Y N N N N N 
2014 [30] 
7. Golombok Rust Inventory of Sexual Y Y Y Y 3 Yi N Y E N N N N 
Satisfaction (GRISS) - 1986 [31] 
8. Jewish General Hospital Sexual Self- N Y N N N N N Y N N N N N 
Monitoring Form ~ 1998 [32] 
9. Derogatis Interview for Sexual Y Y N Y N Y N N N N Uy Y N 
Functioning - Self Report — 1997 
10. Florida Sexual History Questionnaire Y N N N Y Y Y N N N N N 
(FsHQ) - 1991 [33] 
11. Sexual Desire Conflict Scale for 
Women (SDCSW) - 1991 [34] 
12. Brief Index of Sexual Function for Y Y N Y N i N Y i N th N N 
Women — 1994 [35] 
13. Multidimensional Sexuality N N N N N ii N Y Ü Y D N N 
Questionnaire — 1993 [36] 
14. Sexual Function Scale (SFS) - 1988 Y Y vy N N N N Y N N N N N 
B7 
15. Sexual Interaction System Scale — Y Y N N N N N Y N N N N N 
1994 
16. Brief Male Sexual Function. Y Y Y Y N E Y Y N N N N N 
Index/inventory (BMSFI) — 1995 [38] 
17. Sexual Activity Questionnaire (SAQ-F) N N N N v 3 N Y N N N N N 
1996 [29] 
18. Sexual Desire Inventory - 1996 [40] 
19. Sexual History Form (SHF) — 1998 [41] Y Y N N N Y N Y N N N N N 
20. Arizona Sexual Experience Scale Y y N Y N N N Y N N N N N 
(ASEX) - 2000 42] Y Y N i Y N N Y N N N N N 
21, The Female Sexual Functioning Index 
(FSFI) ~ 2003 [43] 
22. Short scale to measure female Y Y N N Y t N N N N uU N N 
sexuality — 2001 [44] 
23. Female Sexual Distress Scale (FSDS) — N N N N N N N N ii N N N N 
2002 [45] 
24. The Sexual Function Questionnaire Y Y N Y Y N N Y N Y N N 
(SQ) 2002 [46] 
25, Sexual Satisfaction Scale for Women N N N N N N N Y Uu N Mi N N 
(SSS-W) - 2005 [47] 
26. Scale for Quality of Sexual Function N Y N N N V N N i N Wr N N 
(QSF) - 2005 [48] 
27, Sexual Arousal and Desire Inventory Y Y N Y N N Y N N N N N N 
(SADI) - 2006 [49] 
28. Cues for Sexual Desire Scale (CSDS] — Y N N N N N Y N N N N Y N 
2006 [50] 
29. Female Sexual Distress Scale Revised Y N N N N N N N E N N N N 
(FSDS-R] - 2008 [51] 
30. Female Sexual Desire Questionnaire. 
(FsbQ) - 2011 [52] 
31. Sexual Distress Scale (SDS) - 2018 
[53] / Sexual Distress Scale Revised (SDS 
R) 


Green (Y) = Yes, dimension covered by the instrument; White(N) = No, dimension omitted by the instrument, 
Light grey = NA,not provided in the reviews 


Green (Y) = Yes, dimension covered by the instrument; White (N) = No, dimension omitted 
by the instrument, Light grey = NA, not provided in the reviews 


Discussion 


Sexual activity is an important part of social functioning and consequently 
QoL. Over 100 self-administered instruments have been developed to respond to an 
interest in assessing sexual function/dysfunction and QoL in the context of research 
and clinical practice. These instruments nevertheless vary in their purpose and 
scope, making selecting the appropriate measurement tool potentially challenging 
for those wishing to assess sexual activity, function and QoL. This meta-review 
aimed to provide an up-to-date overview of generic rather than disease-specific 
instruments that can be used in men and/or women. Thirty-one self-reported mea- 
sures of sexual QoL with documentation of their psychometric properties were con- 
sidered. Only three, however, were developed after 2013, the year the DSM-V 
revised the definition of male and female SD. Furthermore, there seems to be no 
consensus as to what constitutes the key dimensions of sexual QoL beyond the 
phases in the sexual response cycle. Nevertheless, our meta-review highlights that 
there has been an effort to address the specific needs of women, evidenced by the 12 
instruments included which were for use in women [21]. 
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There is a notable lack of instruments designed for use in homosexuals. Only one 
of these reviews sought to explore the availability of instruments in a specific popu- 
lation: gay men. McDonagh’s systematic review from 2014 comprised only seven 
instruments that were potentially viable for use in this specific population. She 
pointed to a number of shortcomings, specifically hetero-normative or heterosexist 
language [22], concluding that there was currently no psychometrically sound mea- 
sure of male sexual function that can be used in gay men. The lack of involvement 
of sexual minorities in the development of these instruments may be at fault. 

The issue of patient/individual involvement in instrument development was also 
raised by Arrington et al. [18] in their review from 2004. They questioned whether 
available instruments would be more comprehensive had items been generated 
using qualitative research in a diverse group of patients either representative of the 
general population or dealing with SD. The study population often selected for 
instrument validation studies is also a concern as many scales appear to have been 
validated in patients with SD, rather than community samples. This raises concerns 
about their ability to detect SD in the general population. 

There is also limited evidence of instruments’ responsiveness or their ability to 
detect change over time. Evidence of responsiveness had only been documented in 
seven of the 31 instruments considered. Users should be wary of this limitation as 
they consider these tools for use in research or clinical practice. 

Surprisingly, none of the reviews addressed the issue of the electronic collection 
of sexual function/QoL in spite of the fact that the practicality of administering, 
mostly face-to-face or self-administered paper-and-pencil, patient-reported out- 
comes (PROs) has been documented as a major barrier to their use, particularly in 
clinical practice. Paper-based approaches require that a physician and/or other staff 
administer the questionnaire during the consultation and enter and/or analyze data 
manually, requiring resources to collect, analyze and utilize PROs data [54]. 
Computerized PRO assessments have become common in settings where PROs 
have been widely adopted, offering a number of advantages over pencil-and-paper 
assessments. For example, applying compulsory items and pre-specifying accept- 
able ranges or values can improve data completeness and quality. Complex skip 
patterns can be programmed or Computer Adaptive Testing (CAT) methods applied 
to ease administration [55]. Immediate data capture also reduces the burden and/or 
costs associated with data entry [56]. Furthermore, from a methodological point of 
view, there is strong evidence to suggest that electronic and paper-and-pencil admin- 
istration methods are equivalent [57]. 

New technologies are yet another means of assessing sexual activity. We see the 
emergence of technologies making use of both active, memory-based data capture 
methods, which are subjective and socially acceptable, and passive, technology- 
based methods—via wearables and personal devices—which are sensory-based, 
momentary, non-judgmental and context-rich. Sexual activity assessment applica- 
tions (also referred to as “sex trackers") take an alternative approach, seeking to 
help individuals improve the quality of their sex lives via various means such as 
educational content including erotica, personalized advice, forums, data visualiza- 
tion, and automated statistics (...). Wac has qualified technologies such as these as 
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“Quality of Life Technologies” (QoLT) or any technology designed for the assess- 
ment or improvement of individuals’ QoL. These “leverage miniaturized comput- 
ing, storage, and communication sensor- and actuator-based, context-rich 
technologies that can be embedded within various personal devices" [58]. Some 
technologies seek to appeal to all, whereas others focus specifically on features that 
appeal to sexual minorities or are deliberately gendered. Applications like Coral? 
(an intimacy app that relies on education), Rosy^ (a platform offering women a 
holistic approach to sexual health and wellness, offering education, self-discovery 
and community), Sex Life? (a sex tracker, sex diary and sex calendar) are novel as 
their core functionalities are not geared towards sexual activity for fertility or the 
prevention of sexually transmitted infections (STIs). Unsurprisingly, most of these 
applications still rely on memory-based approaches to capture users' data, differing 
from now commonplace applications designed to assess and improve physical fit- 
ness. Nevertheless, they offer features that seek to maintain, prevent, or enhance 
sexual QoL, fitting with the aims of QoLT. Many of these application and service 
providers have already begun to collaborate with universities and researchers, often 
establishing cohorts of users and generating a wealth of real-world data. 

Other mobile applications are hybrids, pairing sexual activity assessment with 
the promotion of sexual health (e.g., STI prevention or testing). For example, appli- 
cations like Biem? or Safely’ focus on convenience. The former offers telemedicine 
and at home or laboratory testing. The latter connects users with a network of labo- 
ratories and allows them to receive their results electronically. They have also incor- 
porated interactive features; both allowing users to make their data available to 
partners or potential partners securely and privately and the functionality “Biem 
Connect" allows sexual partners to be notified anonymously if one of their partners 
tests positive for an STI. Nice? is a combination of a sexual activity assessment 
application that allows for some data on STIs to be collected. Data entered through 
these applications can be synched with the Apple HealthKit,? which allows users to 
record their reproductive health data, including the occurrence of sexual activity. 

The use of wearables (e.g. Smartwatches, rings) and personal devices (e.g. sex 
toys) to measure sexual activity appears to be possible as they work by measuring 
whole-body motion (accelerometer), heart rate (photo plethysmography), body 
temperature, and respiration rate. The use of these devices, such as the Oura Ring? 


^? https://getcoral.app/ 

“https://meetrosy.com/ 
Shttps://play.google.com/store/apps/details?id=com.sexdiary.safeforsex 
®http://biemteam.com/ 
Thttps://play.google.com/store/apps/details?id=me.safehealth&hl=fr 
*https://apps.apple.com/us/app/sex-tracker-by-nice/id1 107291612 
?https://developer.apple.com/design/human-interface-guidelines/healthkit/overview/ 
P https://ouraring.com/ 
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or MyMotiv Ring!! (personal health or fitness trackers worn as rings) have been 
used to successfully generate detailed data corresponding to different parts of the 
sexual response cycle. However, most still require users to qualify increases in 
peaks and valleys in physiological data captured by wearable devices as sexual 
activity. Although, to date, the use of fitness trackers and Smartwatches to capture 
this highly sensitive data does not seem to be the approach most favored by QoLT 
developers, there is evidence, mostly from popular culture, that quantified self-ers 
are embracing these new technologies. They are willing to monitor their own behav- 
iors, even their sex lives, to gain a greater understanding of their bodies, in the vein 
of “knowing thyself’ and perhaps even in an attempt to improve their (sexual) QoL 
[59]. Connected personal devices offer yet another avenue for passive data collec- 
tion related to sexual function or activity. For example, users of biofeedback vibra- 
tors like the Lioness!” can, in addition to traditional functionalities, visualize their 
precision sensor-generated data via an application and participate in medical and 
academic research by opting to share their de-identified data. Another connected 
wearable is the Lovely 2.0," a connected penis ring, which, when worn during sex 
learns about your "style" of having sex and provides personalized advice via a dedi- 
cated application. Some research is currently ongoing to validate these sex toys as 
viable research tools. There is a need to understand how data generated from appli- 
cations, wearables and connected devices correlates with direct and indirect mea- 
sures of sexual function and self-reported measures seeking to capture sexual QoL 
as a broader construct. 

As it is important to get information from the best source when standardizing the 
assessment of sexual activity, function and QoL, we proposed the following sources 
of information for each of the different dimensions according to Mayo et al. taxon- 
omy [60] (Table 16.5). These include self-reports like patient-reported outcomes 
(PROs), collected via the proposed self-administered instruments, which clearly 
vary in both length and scope and Ecological Momentary Assessment (EMA) tech- 
niques, in which individuals report states close to the time they experience them, as 
well as other reports like observer/proxy-reported outcomes (ObsRO/ProxyRO) and 
Peer-Ceived Momentary Assessments Methods (PeerMA) [61]. Both ObsRO/ 
ProxyRO and PeerMA would rely on an "other", likely to be an intimate partner 
rather than a friend in the case of sexual activity reporting on an individual's per- 
ceived state and behaviors. The former doing this at fixed time points and the latter 
on a momentary basis, similar to the collection of EMAs. These could be useful 
insofar as externally observable behaviors and states are concerned, e.g. perceived 
desire, arousal, plateau, and orgasm as well as the frequency of sexual activity with 
a partner. Finally, technology-reported outcomes (TechROs) which could be 


! https://mymotiv.com/ 
P https://lioness.io/ 
P https://ourlovely.com/ 
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collected via wearables, Smartphone behavioral metrics, or connected devices could 
be used for many of the domains important to the assessment of sexual function/ 
QoL, namely if whole-body motion and physiological parameters could be co-cali- 
brated with valid direct, indirect, or self-reported measures of sexual function/QoL, 
We have summarized some options for assessing and measuring different domains 
below in Table 16.5. 


Conclusions 


There are a number of generic self-reported instruments that can ease the collection 
of data on sexual function and QoL in both research and clinical practice. The 
emerging use of sexual activity assessment applications, wearables and personal 
devices, may also provide another less invasive avenue for the collection of sexual 
activity data. There is nevertheless a need to understand the measurement properties 
of these novel means of data generation as well as users’ willingness to submit or 
share their highly personal data with researchers or providers in order to facilitate 
their adoption in research and clinical practice. There is still a long way to go if we 
wish to exploit the full potential of QoLT for sexual health and QoL. 
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Chapter 17 A 
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Introduction 


An individual’s financial resources are directly related to their ability to meet cur- 
rent and future needs. Higher levels of financial assets and lower debt have been 
found to be positively associated with financial satisfaction [1, 2]. Findings from the 
2012 National Financial Capability Survey report that the presence of an emergency 
fund has been found to be positively associated with financial satisfaction [3]. On 
the other hand, inadequate financial resources can lead to financial strain and finan- 
cial distress. This financial burden has been associated with decreased treatment 
adherence, worsened mortality, increased bankruptcy. Financial strain can have a 
powerful impact on a person’s perception of their overall well-being. At the same 
time, the growth of personal finance apps and technology-enabled tools have 
exploded over the past 20 years with the potential for objective, quantitative assess- 
ments, and self-management of financial resources. Although financial resources 
have been shown to have a direct effect on quality of life, few studies have addressed 
the impact of financial resources and financial burden on quality of life and the role 
of QoL technology-enabled tools for measuring and managing financial resources 
and improving quality of life. 


Definition of Financial Resources 


According to the WHOQOL theoretical model, quality of life encompasses several 
key domains: Physical Health, Psychological Health, Social Relationships, and 
Environment. One of the environmental facets of quality of life is financial resources. 
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The facet of financial resources explores a person’s view of how his/her financial 
resources and the extent to which these resources meet the needs for a healthy and 
comfortable lifestyle, and what the person can afford or cannot afford which might 
affect quality of life. This facet includes a sense of satisfaction/dissatisfaction with 
those things which the person’s income enables them to obtain and the sense of 
dependence/independence provided by the person’s financial resources (or 
exchangeable resources), and the feeling of having enough regardless of the respon- 
dent’s state of health or whether or not the person is employed. It acknowledges that 
a person's perspective on financial resources as “enough,” “meeting my needs,’ etc. 
is likely to vary greatly [4]. 

Originally, financial well-being was seen as synonymous with material and 
objective financial resources (e.g., income) and was investigated at the country 
level, without including individuals’ perceptions. However, once Easterlin [5] rec- 
ognized the importance of subjective perceptions of financial well-being, it has been 
mainly studied at the individual level. 

Nowadays, scholars agree that financial well-being has both an objective and a 
subjective side, believing that objective financial well-being consists of individuals’ 
material financial resources (e.g., income), whereas subjective financial well-being 
is individuals’ perception and cognitive and emotional evaluation of their own 
financial condition [6]. This subjective side frequently has been investigated mainly 
in populations considered critical or atypical from a financial point of view, such as 
the elderly [7], unwed women [8], and divorced women [9]. 

Although financial resources have been shown to have a direct effect on quality 
of life, few studies have addressed the impact of financial resources and financial 
burden on quality of life and the use of personal, quantitative methods and technolo- 
gies to assess the financial resources and the financial well-being of individuals. 

This chapter reviews the literature about (1) the effects of financial resources and 
financial burden on treatment outcomes and overall quality of life; (2) the state-of- 
art tools for measuring financial resources by individuals and financial and health 
professionals; (3) the evaluation of Web-based interventions for enhancing financial 
resource management; and (4) the behavioral and technology-related factors for 
successful adoption of QoL technology-enabled methods and financial resource 
management tools for improving individual life satisfaction and financial well-being. 


Current Research About Financial Resources and Financial 
Well-Being 


Financial Resources and Financial Satisfaction 


Researchers over the past 30 years have examined both objective and subjective 
measures to describe the financial condition of individuals and families. Whereas 
objective indicators have been used to predict one’s perceptions about their financial 
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condition, such indicators do not measure the depth of one’s feelings about or reac- 
tion to their financial condition. Several researchers have examined factors contrib- 
uting to psychological well-being and found economic distress to be a good predictor 
of lower levels of well-being [10, 11]. Mills [10] found that a key determinant of 
psychological well-being was the level of economic distress reported. Rettig and 
Danes found that people who perceived their income to be inadequate to meet even 
basic living expenses reported experiencing negative feelings and lower satisfaction 
with the perceived gap between their standard and their level of living. Such normal, 
negative reactions to adverse economic condition can reduce individuals’ psycho- 
logical well-being [12]. 

An individual’s financial resources are directly related to their ability to meet 
current and future needs. One theory argues that relative income (both to others 
and to previous periods) rather than absolute income may be important in explain- 
ing variations in life satisfaction, i.e., this includes income relative to others, which 
impacts an individual’s status in society, and relative to the individual’s income in 
previous periods, which impacts one’s habits and view of what is the norm [13]. 
Boyce [14] found that the ranked position of an individual’s income predicts gen- 
eral life satisfaction, whereas absolute income and reference income have no effect. 
Similarly, the rank of a person’s income or wealth within a social comparison 
group, rather than income or wealth themselves or their deviations from the mean 
within a reference group, is more strongly associated with depressive symp- 
toms [15]. 

Researchers generally agree that subjective well-being (SWB) tends to be higher 
among people with abundant economic resources as compared to people with lim- 
ited economic resources [1, 2]. However, recent research has begun to distinguish 
two aspects of subjective well-being: emotional well-being which refers to the emo- 
tional quality of an individual’s everyday experience that make one’s life pleasant or 
unpleasant, and life evaluation which refers to the thoughts that people have about 
their life when they think about it. Analyzing results from the Gallup-Heathways 
Well-being Index and Cantril’s Self Anchoring Scale, Kahneman [16] examined 
whether money buys happiness separately for these two aspects of well-being. 
While life evaluation rises steadily with increases in income, emotional well-being 
also rises but there is no further progress beyond an annual income of USD75,000. 
High income buys life satisfaction but not happiness; low income is associated both 
with low life evaluation and low emotional well-being. 

Although the literature on the relationship between economic standing and SWB 
has become substantial, only a limited number of studies, to date, have focused on 
this relationship in mid-life and old age [1, 2]. In general, the findings of the studies 
on the wealth-SWB relationship have demonstrated that greater wealth leads to 
higher SWB. Some studies have also demonstrated that the effect of wealth is stron- 
ger than the effect of income. Wealth refers to the stock of assets (e.g., savings, real 
estate, businesses) less their debt held by a person or household at a single point in 
time vs. income which refers to money (e.g., wages, rents on property, government 
assistance) received by a person or household over some period of time [2]. It should 
be noted, however, that the magnitude of the association between wealth and SWB 
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may depend upon the welfare regime and the degree of social support (including 
type of pension) provided by the state. Similarly, those with more economic 
resources are more likely to cope with stressful events when they occur. Most stud- 
ies on the issue were conducted within a cross-sectional research design, very few 
studies have addressed the wealth-SWB relationship from a longitudinal perspec- 
tive taking into account the dynamics of household wealth and life transition events 
over time [1, 2]. 


Longitudinal Perspective of Financial Resources and Financial 
Well-Being 


Prior research consistently has found that older adults, despite low incomes, are 
more financially satisfied than younger adults. This “satisfaction paradox” is typi- 
cally attributed to elders’ supposed psychological accommodation to poor financial 
circumstances. However, data from the first wave of the Norwegian NorLAG study 
shows that material circumstances are more important to the financial satisfaction of 
the elderly. A considerable part of the higher financial satisfaction with increasing 
age can thus be explained by greater assets and less debt among the elderly. 
Nonetheless, assets and debt do not mediate this relationship at lower incomes, 
because older people with little income have very little accumulated wealth. Older 
adults with little income and wealth have a much stronger tendency to be financially 
satisfied than younger, equally poor counterparts. An “aging paradox” remains in 
this field [1]. 

A life-course perspective attempts to place indicators of financial well-being in 
context, yet it is not widely used in the financial well-being literature. Studies that 
do use a life-course perspective tend to focus on the ‘U’ shaped gradient of financial 
satisfaction over time [1], arguing that lower debt and greater asset wealth explain 
why older people are more financially satisfied despite having lower incomes than 
younger people. Employment-based income was a stronger determinant of younger 
people’s financial satisfaction whereas investment income was a stronger determi- 
nant for older adults [2]. 


Financial Burden and Its Effect on Treatment Outcomes 
and Quality of Life 


The financial burden of care, especially for cancer, is slowly becoming more recog- 
nized among providers, institutions, and policymakers. The high costs of cancer 
care, even for patients with health insurance, can lead to poorer outcomes for 
patients, decrease quality of life, heighten stress, and, in some cases, increase mor- 
tality rates [17]. 
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According to Fenn [17], increased financial burden because of cancer care costs 
is the strongest independent predictor of poor quality of life among cancer survivors 
over the age of 18. Patients who reported “a lot" of financial problems were approxi- 
mately four times less likely to report a quality of life that was “good” or higher 
compared with patients who reported no financial problems. The magnitude of 
cancer-related financial difficulty was a more significant predictor of quality of life 
than age, education, race/ethnicity, and family income. These findings highlight the 
potentially powerful impact of financial strain on a patient's perception of their 
overall well-being after a cancer diagnosis. 

Financial resources can act as a buffer for subjective well-being after the onset of 
a disability or disease. Using data from the Health and Retirement Study, Smith [18] 
found that wealth measured prior to the onset of a disability protected participants’ 
well-being from some of the negative effects of a new disability. Participants who 
were above the median in total net worth reported a much smaller decline in well- 
being after a new disability than did participants who were below the median. There 
was also some evidence that the buffering effect of wealth faded with time, as 
below-median participants recovered some of their well-being. 

Regarding frailty, financial resources, and subjective well-being in later life, 
Hubbard [19] found that those with greater financial resources reported better sub- 
jective well-being with evidence of a “dose-response” effect. The poorest partici- 
pants in each frailty category had similar well-being to the most well-off with worse 
frailty status. Hence, while the association between frailty and poorer subjective 
well-being is not significantly impacted by higher levels of wealth and income, 
financial resources may provide a partial buffer against the detrimental psychologi- 
cal effects of frailty. 


Standardized Self-Report Based Measurements of Financial 
Resources and Quality of Life 


Two different approaches exist to measure financial resources and financial well- 
being. One approach utilizes a standardized instrument for self-assessment and is 
frequently used by physical and mental health professionals. 

One standardized instrument, The InCharge Financial Distress/Financial 
Well-Being Scale (IFDFW), is an 8-item scale designed to measure a latent con- 
struct representing responses to one's financial state on a continuum ranging from 
overwhelming financial distress/lowest level of financial well-being to no financial 
distress/highest level of financial well-being. The robust Cronbach's alpha of 0.956 
for the IFDFW indicates high internal consistency, and factor analysis indicates 
measurement of one factor, verifying that the indicators together estimate only one 
latent construct. Thus, the IFDFW Scale provides a high level of confidence for 
researchers and practitioners using the scores to indicate perceived levels of finan- 
cial distress/financial well-being in individuals and groups of consumers [20]. 
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This scale has been used to develop and deliver high quality, effective workplace 
financial programs [21]. However, the scale was developed to measure financial 
distress and well-being in general and may not include factors relevant to a specific 
stage of life such as emerging adulthood [20]. The IFDFW is also known as the 
Personal Financial Well-Being (PFW) scale. 

Another standardized instrument, The Multidimensional Subjective Financial 
Well-being Scale (MSFWB) is a 25-item scale measuring five different aspects of 
subjective financial well-being (general subjective financial well-being, money 
management, peer comparison, having money, financial future) with acceptable 
psychometric properties and was developed for use with emerging adults in 
European countries namely Italy and Portugal. Validity of its 25 items was assessed 
through consultation with experts on the construct as well as the emerging adults 
themselves (qualitative research). Furthermore, the functioning of these items was 
confirmed by quantitative results showing the stability of the scale’s factorial struc- 
ture, its generalizability across different emerging adult subgroups, its relationship 
with convergent and criterion-related measures, and its internal consistency. 
However, the generalizability of this measure across European countries and use in 
the US has not been demonstrated [22]. 


State-of-the-Art Apps and Tools for Measuring Financial 
Resources and Financial Well-Being 


Use of Personal Finance Apps and Financial Resources 


The second approach to measure financial resources and financial well-being uti- 
lizes Artificial Intelligence (AI) and algorithms and is frequently used by financial 
advisors and individuals interested in the self-management of their financial 
resources. This approach can range from budgeting and personal financial apps to 
more sophisticated and complex software programs used by financial advisors and 
robo advisors. More recently SigFig and Jemstep offer QoL technology-enabled 
investment and financial management tools. 

Smartphone personal finance apps have grown significantly with the growth of 
smartphone users. According to a recent (February 2018) Bankrate.com Report [23] 
about the use of finance-related apps, including those from traditional banks and 
fintech players, 63% of US adults who use a smartphone have at least one financial 
app. The average smartphone user has 2.5 financial apps. Among millennials (ages 
18-37), the average is 3.6. It drops to 2.3 for Gen X (ages 38-53) and 1.4 for Baby 
Boomers (ages 54—72). The most common financial apps are full-service banking 
apps; 55% of US adults who have a smartphone have at least one full-service bank- 
ing app; 23% have at least two. Peer-to-peer payment (P2P) apps (such as Venmo, 
PayPal and Square Cash) are the second most common. Forty-one percent of smart- 
phone users have at least one of these and 15% have more than one. Standalone 
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budgeting and investing apps are less common; they can be found on just 1896 and 
1796 of smartphones users, respectively. Mint, Clarity Money, and Wally are among 
the best-known budgeting apps. Stash, Acorns, and Betterment are popular for 
investing. People with banking apps are the most engaged. Seventy percent say they 
use them at least once a week. Fifty-six percent of budgeting app users, 51% with 
investing apps, and 38% with P2P payments apps reported likewise. 

Consumers who avoid mobile banking tend to fall into two camps: people who 
like their routine and people who are afraid of fraud. Twenty-nine percent of finan- 
cial app users believe financial apps are better than non-financial apps versus just 
9% who think they are worse. 

Millennials are the most divided generation on this issue: 3196 of them say finan- 
cial apps are better, but 1596 say they are worse. Regular use of mobile banking apps 
has potentially significant implications for the financial health of users. Using 
mobile banking makes the user more aware of his/her current financial status and 
impacts his/her financial satisfaction. 


Types of Personal Financial Management Apps 


According to Investopedia [24], some of the best personal finance apps for manag- 
ing finance resources in 2020 are the following apps. Each of them relies on embed- 
ded sets of models and algorithms that maximize some predefined goal like track 
past spending and saving habits or plan a future budget or invest excess change or 
reminder of payment due or improve money and investment management or moni- 
tor credit score or keep track of income and expenses for multiple individual 
projects. 

The Best Money Management App is the Mint.com from Intuit. It is a free all-in- 
one resource for creating a budget and tracking spending which can be linked to 
bank accounts giving a snapshot of one's financial situation momentarily and over 
time. It creates budgets based on previous spending habits and alerts users if they 
are over spending limits. It also offers a financial goals section that assists in identi- 
fying and tracking goals for almost any expenditure [25]. Similar websites are 
MoneyStrands.com, PearBudget.com, Wesabe.com, JustThrive.com 

The Best Budgeting/Debt App is You Need a Budget (YNAB). YNAB is built 
around a simple principle that every dollar has a job in a personal budget, be it for 
investing, for debt repayment, or to cover living expenses and forces an individual 
to live within his/her actual income. If one gets off track, YNAB helps one to see 
what can be done differently to balance the budget. There is also a built-in account- 
ability partner. EveryDollar Budgeting is a similar zero-based budgeting app which 
tracks spending and plans for purchases. PocketGuard, another alternative, builds a 
personalized budget based on your income, bills, and goals. 

The Best Tracking Expenses App is Wally. Wally helps the user to organize and 
track personal and professional expenses. Wally lets the user scan receipts and add 
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notes allowing the user to see where money has been spent [26]. Wally uses machine 
learning and AI algorithms to adapt to user preferences and behavior. 

Best App for Easy Saving is Acorns. Acorns is a useful app for inexperienced 
investors. 

It links to a user's bank account and will make investments automatically or 
manually [25]. It takes the user's extra change from transactions and automatically 
directs it to a savings account after evaluating the user's spending and income his- 
tory [27]. A similar website is Stockpile. 

Best App for Freelancers is Tycoon. This app is perfect for freelancers and those 
who are self-employed and get paid at the completion of a contract. It standardizes 
the details of a contract, puts in a timetable for it, and keeps track of payments that 
have come in, are scheduled to come in, or that are past due. It makes it easy to see 
which clients have not paid yet. 

App users report that these personal finance tools simplify the formerly confus- 
ing and complicated investment and financial management process into something 
approachable, learnable, and scalable, but the question arises if they are making the 
users better at managing his/her money or encouraging the user to spend more. 
Money apps can have a positive and a negative effect. They can make an individual 
more aware of how much one is spending and help one stick to a budget, but the 
convenience of contactless payment could also be encouraging even more spend- 
ing [28]. 

Unlike the standardized instruments for measuring financial well-being, none of 
these personal finance apps have been validated, although some research in behav- 
ioral sciences provide examples of the importance of financial self-tracking on indi- 
vidual's spending and saving behaviors [29, 30]. A new class of applications and 
web sites called personal informatics is appearing that collects behavioral informa- 
tion about users and provides access to this information to help users become more 
aware of their own behaviors. Personal informatics systems support users in under- 
standing various aspects of their life, behaviors, habits, and thoughts. They help 
users build self-understanding by providing a means to collect personal history, as 
well as tools for review or analysis. Increased self-understanding has many benefits, 
such as fostering insight, increasing self-control, and promoting positive behaviors 
such as energy conservation and financial management [30] (Figs. 17.1, 17.2 
and 17.3). 


Web-Based Technology Tools for Financial 
and Investment Analysis 


Web-based tools are useful for long-term planning and financial and investment 
analysis. These tools measure the individual's attained level of financial resources 
and then estimates the future level of financial resources based on certain assump- 
tions about investment interest rates, inflation rates, tax brackets, the amount and 
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Fig. 17.1 Screenshot of You Need a Budget (YNAB) app 


savings rate through employer-based plans plus individual savings, expected retire- 
ment age, and major funding needs such as college funding, home purchase, finan- 
cial needs for incapacitated relative (such as a child or parent), major medical 
expenses (such as cancer or a chronic condition), or for a legacy to a charity (such 
as one's university). This approach attempts to provide an objective, quantitative 
assessment of financial resources needed to attain these financial goals and needs. 
Individuals can then assess the gap between their future financial needs and their 
estimated financial resources to develop strategies to close the gap. Often this is 
done with the aid of a financial advisor; hence many of the software programs and 
technology developed to serve this function have been developed by investment 
management companies. 

More recently automated investing services also known as robo advisors have 
emerged. Robo-advisors are digital platforms that provide automated, algorithm- 
driven financial planning services generally with little or no human intervention 
used to automatically allocate, manage, and optimize a client's assets and invest- 
ment portfolios. 

One tool, Personal Capital, is an investment management service for tracking 
wealth and spending that combines the algorithms used by robo advisors with 
access to human financial advisors. While Personal Capital is primarily an invest- 
ment tool, its free app includes features 

helpful for budgeteers looking to track their spending. It allows one to connect 
and monitor checking, savings, and credit card accounts, as well as IRAs, 401(k)s, 
mortgages, and loans. The app provides a spending snapshot by listing recent trans- 
actions by category and displays the percentage of total monthly spending by each 
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Fig. 17.2 Screenshot of 
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category. Personal Capital also calculates and tracks user’s net worth and provides 
a breakdown of one’s investment portfolio [22]. 

Perhaps the most evident area where technology has impacted the world of 
investing is in the capabilities to analyze investments and develop investment strate- 
gies. Ranging from “the basics” at Yahoo Finance and Google Finance to the more 
advanced analytics like Morningstar and Bloomberg terminals, to a host of other 
specialized sites for investment analysis, the capabilities for investment analysis 
have come a long way. 

A more recent crop of tools like SigFig and Jemstep are also making it easier for 
consumers to analyze an existing portfolio and immediately get actionable advice 
about what can be improved. The latest technology evolution disrupting the world 
of investing uses robo advisors—services like Betterment and Wealthfront—that 
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are affordable and will construct the entire asset-allocated passive strategic portfolio 
for the investor. The depth to the portfolio construction process also makes it feasi- 
ble to leverage technology tools for additional investment value-adds that were pre- 
viously done in a more arduous manual processes, from proactive tax loss (or gains) 
harvesting to supporting good asset location decisions. Such services can now be 
offered effectively directly to consumers and easily implemented via robo advisors 
(through advisors and their “rebalancing” and trading software tools) [22]. 

Robo advisors have a couple of key advantages. Because they do not have to 
employ many people, they typically have lower fees and lower account minimums 
than traditional investment advisors. Some robo advisors like Betterment, 
WiseBanyan, and Bloom even have no account minimums. Of course, robo advisors 
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Fig. 17.4 Screenshot of Personal Capital Investment Management desktop app 


have drawbacks. For one thing, the user cannot rely on a robo advisor to do compli- 
cated financial planning or to give him/her legal advice. 

Moreover, a robo advisor will not be able to give the user the kind of nuanced 
financial advice that a human being might be able to provide. 

As more and more tools become accessible directly to consumers, many of the 
services once provided by financial advisors to analyze, review, and construct client 
portfolios can now be done at a dramatically lower cost and without the advisor 
being involved at all. These apps and tools have continued to commoditize many 
financial resource management functions previously performed by stockbrokers 
and financial advisors. There has been specifically a rapid growth of the number of 
personal finance apps and app users and an increase in the amount of monies 
directed towards these apps, but there has been little or no empirical studies address- 
ing the direct impact of these apps on financial resources and the financial well- 
being of users (Fig. 17.4). 


Other Web-Based Interventions and Strategies 


According to Statista, as of February 2020, Yahoo Finance, MSN Money Central, 
and CNN Money are the three most visited investment and finance websites in the 
US with 70 million monthly visits, 65 million monthly visits, and 50 million 
monthly visits respectively) [31]. 
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Personal finance websites now attract one-in-four people who use the Internet, 
according to comScore.com which tracks Internet traffic, rivaling Facebook in pop- 
ularity [32]. Currently, the financial capacity in the US is reported by FINRA 
Investor Education Foundation [33]. 

There are three categories or models of Web-based personal finance websites 
which can be of use in enhancing one's financial assets and financial well-being: 
Financial Data Aggregators, Financial Decision Aids, and Financial 
Communities [34]. 

Financial Data Aggregators are highly effective tools which use the computer’s 
power to compile and analyze massive amounts of data harnessing computing 
power to perform complex analyses that few individuals would have the time, inter- 
est, or ability to perform on their own. 

Aggregators are effective as personal finance tools because they continually ana- 
lyze the underlying data as it changes. As a result, aggregators keep individuals 
coming back for fresh information, which, in turn, keeps users engaged and focused 
on their financial affairs. 

Aggregators also encourage individuals to analyze changes, over time, in their 
finances. These tools apply the same format to the data each time, making it easy to 
compare progress over months or years. These tools, therefore, provide a frame- 
work for a richer understanding of one’s financial profile. 

Financial Decision Aids are effective because they calculate costs and benefits 
that range from simple mortgage calculators to complex online financial planning 
decisions. By requiring individuals to input their unique data and other personal 
information, individuals can generate personalized results that illuminate an impor- 
tant element of financial health. 

Financial Communities enable individuals to meet and discuss their personal 
finances and provide support and advice to each other, compete in financial games, 
swap coupons, or attend virtual parties focused on personal finance. Even personal 
finance blogs are, in one way, communities and are categorized as such in this report. 


Use of Web-Enabled Tools to Increase Financial Literacy 
and Financial Well-Being 


Web-enabled tools can be used to increase financial literacy, financial well-being, 
and financial satisfaction in later life. The Consumer Financial Protection Bureau 
(CFPB) publishes financial information to increase financial literacy. They advocate 
that the ultimate measure of success for financial literacy efforts should be improve- 
ment in individual financial well-being [32]. 

According to the Consumer Financial Protection Bureau (CFPB), financial well- 
being can be defined as a state of being wherein individuals: 


* Have control over day-to-day, month-to-month finances; 
* Have the capacity to absorb a financial shock; 
e Are on track to meet their financial goals; and 
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* Have the financial freedom to make the choices that allow them to enjoy life. 


Personal finance apps can provide users with personalized financial information for 
establishing and monitoring budgets, spending and saving habits, progressing 
towards financial goals, etc. 

Because individuals value different things, traditional measures such as income 
or net worth, while important, do not necessarily or fully capture this last aspect of 
financial well-being. 

Joo and Grable [35] advocate for the development and testing of a framework for 
understanding the determinants of financial satisfaction. Direct, as well as indirect, 
effects on financial satisfaction were identified using a path analysis method. It was 
determined that financial satisfaction is related, both directly and indirectly, with 
diverse factors including financial behaviors, financial stress levels, income, finan- 
cial knowledge, financial solvency, risk tolerance, and education. Findings support 
the continued and increased use of targeted education initiatives directed at improv- 
ing the financial literacy and behavior of family and consumer economics constitu- 
encies. Quality of life technologies can support these initiatives. 

However, although there are a tremendous number of personal finance apps and 
tools, relatively little has been published on a causal relationship between financial 
knowledge and financial behavior. Empirical studies of the effectiveness of these 
personalized QoL technology-enabled tools and interventions in increasing finan- 
cial resources and enhancing financial well-being needs to be done. 


Operationalizing Self-Report Measurements of Financial 
Resources and QoL 


In this subsection we reflect how the above-mentioned tools could help to opera- 
tionalize data collection for self-reports like IFDFW and MSFWB (defined earlier). 

For the IFDFW, the data collected via these technology tools could enable the 
use of an analytics approach to questions ranging from "Living on paycheck-to- 
paycheck" basis (in case one's account is reaching low at the end of the month) to 
"Ability to handle $1000 financial emergency" e.g., by analyzing all the available 
assets and the saved amounts for at least $1000 available to an individual. Questions 
such as “Worry about being able to meet normal monthly living expenses" can be 
diagnosed and remediated through the creation and monitoring of budgets and 
establishment of alerts when approaching overbudget limits utilizing budgetary 
apps. "Ability to manage money" can be assessed through data aggregation from 
various banking, credit card, and investment accounts and comparison to targets 
utilizing money management and investment tools. 

For the MSFWB which consists of five different aspects of subjective financial 
well-being— general subjective financial well-being (GS), money management 
(MM), peer comparison (PC), having money (HM), and financial future (FF), 
numerous questions can be operationalized using data collected from technology 
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tools. For questions about the having money aspect of subjective well-being such as 
"At times I do not have the money to buy what I need,” “I cannot do some things 
with my friends, because I do not have the money to do them,” and "Sometimes I 
miss the cash to buy things I need;" personalized tools can be used to monitor spend- 
ing habits against predetermined budgets and setting alerts if going over pre-set 
spending limits. For questions about evaluating issues such as "Satisfaction with the 
way I manage my financial situation,’ budget and money management tools can 
provide spending snapshot of recent transactions by category and displays the per- 
centage of total monthly spending by each category so that users can evaluate their 
money management performance against predetermined targets. Data can be indi- 
vidually analyzed or aggregated across multiple categories—credit card, banking, 
and investment management. Financial future concerns about "In the near future, 
having enough money to carry out my plans" can be operationalized by using apps 
to set up separate budgets for personal expenses from special budgets for future 
plans, contractual assignments, or multiple projects with different timetables and 
overlapping income and expenses. These tools are ideal for self-employed, gig 
workers or freelancers. By using robo advisors or investment management web- 
based tools and software with some advisory support and comparative investment 
data resources, a user can answer whether an individual's “Financial situation is 
better [or worse] than that of one's peers." 

The use of quality of life technologies together with standardized self-report- 
based measurements of financial resources can serve as predictors of level of cur- 
rent and future financial well-being and financial distress and an individual's quality 
of life. 


Success Factors for Technology Adoption 


Computer self-efficacy and anxiety, ability to solve novel reasoning problems (fluid 
intelligence) and remembering and using previously acquired information (crystal- 
lized intelligence), attitudes and experience with technology and the World Wide 
Web are important predictors of the use of QOL technology-enabled tools [36]. 
Millennials have often led older Americans in their adoption and use of technology, 
and this largely holds true today. But there has been significant growth in tech adop- 
tion since 2012 among older generations—particularly Gen Xers and Baby Boomers 
[37, 38]. Older users may not adopt a new tool or technology merely because it is 
available. They must also perceive how the system is personally useful compared to 
existing methods [39] and that it is easy to use. As prior research has shown [40], 
older adult's adoption of new technologies may critically depend on whether they 
understand the costs and benefits of those technologies (perceived utility). Previous 
Pew Research Center surveys have found that the oldest adults face some unique 
barriers to adopting new technologies—from lack of confidence in using new tech- 
nologies to physical challenges manipulating various devices [39]. 
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On the other hand, higher evaluation of life quality was found significantly cor- 
related with the number of new media technologies owned. Owning and using new 
media technology seems to have become a part of defining one’s lifestyle that 
emphasizes enjoying oneself and enjoying life. Finally, the use of both traditional 
and new media, except for TV watching, was found complementary in improving 
people’s living quality [41]. 

The use of tools for financial management definitely falls into the domain of use 
of new media technologies, where the gains may be substantial, while, at the same 
time, the cost of the use may be literal. Namely, the user may lose not only time to 
learn and operate the tool but may also lose his/her own money, e.g., because of the 
use of inadequately configured tools. The success for technology adoption in the 
financial domain is therefore multidimensional, and interdependent and shall be 
researched further. 


Discussion and Conclusions 


Around the world, the financial landscape is becoming increasingly complex. In 
response to this, an array of Web-based and non-Web-based technology initiatives 
seek to address financial participation, education, and inclusion for the broad popu- 
lation, as well as, for vulnerable populations. There is an increased expectation that 
these tools and programs will improve the financial well-being of individuals and 
households. But financial well-being is inadequately conceptualized and inconsis- 
tently defined, making it difficult to understand and improve financial outcomes and 
assess financial resources. Existing conceptualizations do not adequately account 
for the dynamic interplay between a person’s environment and their financial well- 
being as well as how aspects of financial well-being can interact according to age, 
life stages, and culture. 

The recent Quantified Self Movement (QS) and the wealth of digital data origi- 
nating from wearables, applications, and self-reports is enabling QS practitioners to 
better address the diverse domains of daily life—physical state, psychological state, 
social interactions, and environmental context which contribute to an individual’s 
Quality of Life (QoL). This systemic monitoring approach of assessing an individ- 
ual’s state and behavioral patterns through these different QoL domains on a con- 
tinuous basis can provide real-time performance optimization suggestions [42, 43]. 

As the Quantified Self Movement (QS) expands and the availability of quality- 
of-life technologies grow, the incorporation and self-monitoring of financial 
resources will become an increasingly important component of an individual’s 
overall quality of life [42, 43]. In the future, we may see wearable or digital track- 
ing devices that encourage both fitness goals (steps taken, calories burned, hours 
slept) and financial goals (interest earned, fees avoided, milestones reached). This 
is one way that financial resources can play a role in the “quantified self"—by 
encouraging financial fitness. Second, inventive thinking as the intersection of 
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fintech and healthtech can create incentives and new categories for consumers to 
blend sound physical and financial habits for overall well-being. The self-tracking 
of quantifiable financial data and qualitative data, e.g., mood, stress, depression, 
happiness, productivity can enable individuals to monitor changes in their financial 
behaviors based on changes in physiological and psychosocial factors such as the 
correlation between spending behaviors and emotional state. Third, self-tracking 
and monitoring with QS devices and related apps may lead to more proactive than 
reactive financial behaviors by providing users with information on their credit 
card and debt management and the financial performance analysis of investment 
portfolios. Lastly, in personal finance decisions, future wearable devices may be 
able to nudge the wearer with a gentle vibration, that they just should not buy the 
product that they are looking at and instead save the money for a more important 
goal thus leading to better self-management of financial resources and life 
satisfaction. 

An important aspect of the self-tracking of financial resources is that QS finan- 
cial activity fundamentally includes both the collection of objective metrics data 
and the subjective experience of the impact of this data. Self-trackers have an 
increasingly intimate relationship with QS data as it mediates the experience of real- 
ity. In self-tracking of financial resources, the cycle of experimentation, interpreta- 
tion, and improvement transforms the quantified self into an improved “higher 
quality” self. The quantified self provides individuals with means for qualifying 
themselves, through which some higher level of financial performance and financial 
resources may be attained or exceeded leading to greater financial well-being and 
life satisfaction. 

In the case of health impairment, the individual’s financial situation and its rela- 
tionship with life quality is even more complex. Despite the limited number of 
qualitative and quantitative studies available, likely owing to the relative infancy of 
the field, financial resources and the financial burden of treatment seems to play a 
critical role in quality of life. In particular, the use of the QS tools and methods 
presented here can better integrate the costs of care and available financial resources 
when coordinating treatment while attempting to enhance the patient’s treatment 
outcomes and quality of life. 

Thus, the use of QoL technology-enabled apps and methods can play an impor- 
tant role in the overall quantification and self-management of financial resources, 
the reduction of financial burden and distress, increased financial literacy, and the 
enhancement of financial well-being and financial satisfaction. Ideally, the use of 
these QoL technologies should be evidence-based and designed and developed to 
enhance the effectiveness, efficacy, and appropriateness of their use and based on an 
understanding of factors that influence acceptance by users. Greater development 
and integration of quality of life technologies can help address the need for better 
self-management of financial resources for short-term and long-term goal attain- 
ment, and the enhancement of the quality of life of an individual and families 
at large. 
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Part V 
Environment 


Chapter 18 A 
Artificial Intelligence and Quality of Life: giv 
Four Scenarios for Personal Security 

and Safety in the Future 


Sylvaine Mercuri Chapuis 


Introduction 


At the beginning of the 2000s, international institutions such as the European Union 
invested heavily in artificial intelligence (AI; [1]). Academic studies of AI started 
appearing within the decade. Later on, researchers began studying the implications 
of AI for quality of life (QoL). They focused on individuals’ attitudes and motiva- 
tions regarding the use of AI, as well as the behaviors, and practices of AI use (or 
non-use) they engaged in. Studies such as those carried out by Wright [1] and 
Wright et al. [2] quickly drew attention to the potential benefits and drawbacks of 
AI use, discussing its potential for fostering economic growth, convenience, secu- 
rity, and individual and social safety, as well as growing concerns over reduced 
privacy, profiling, surveillance, spamming, and identity theft or fraud, among other 
issues. In relation to freedom, a key factor in QoL, the emergence of AI has coin- 
cided with the rapid development of the Internet and associated digital technologies, 
effectively erasing the physical borders between individuals and granting them new 
freedoms. During this time, various questions around the issue of AI have emerged, 
including ones related to the quantification of “thought” and the protection of the 
individual online. 

Non-academic literature has often discussed AI and its implications for QoL, 
since technological developments over the past decades have created new needs for 
the average individual, while researchers have continued to develop new 
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applications of AI to fill those needs. For example, Pew Research Center,' a non- 
partisan fact tank, has revealed several benefits and risks of AI use while stimulating 
open discussions about AI, the Internet, and the future of social relations and human- 
ity, including the visions of the millennial generation, and the disruption of estab- 
lished business models. 

At present, there are still gaps in the research concerning the social impact of 
AT's emergence, particularly regarding individuals’ attitudes and beliefs about AI 
and its actual use. There are vast differences between the way AI use is popularly 
imagined and how it is really used, and these gaps only widen when one considers 
future trends. This chapter aims to bridge this gap by presenting the results of a 
survey of 1000 university students from Western Switzerland regarding how they 
imagine the future of AI use and its implications for technologies and QoL, espe- 
cially in relation to personal security and safety. 

After a brief outline of the state of the art of AL, we present a tool called 
"Futurescaper" designed to animate collective reflection and foresight. We then 
present the methodology that was used to collect data in this study, along with the 
four scenarios for the future developed using the Futurescaper tool according to 
students’ responses, which are called, respectively, “Al for the best,’ “AI down,” "AI 
for business," and "AI freeze." These scenarios may serve as starting points to enrich 
discussions on AI and QoL. It should be noted that in this study, we do not consider 
questions concerning wealth, employment, the environment, physical and mental 
health, education, recreation and leisure time, social belonging, or religious beliefs 
that factor into QoL. Instead, the study focuses on questions on safety and security 
in relation to QoL. 


State of the Art of AI and Questions for the Future 


Artificial intelligence (AI) refers to a quality of systems that rely on datasets typi- 
cally originating from a range of connected objects (e.g., computers, tablets, smart- 
watches, and connected wristbands) and that are individually and collectively 
capable of quickly analyzing data implementing search, pattern-recognition, learn- 
ing, planning, and induction processes. Artificial intelligence has considerable 
power [3] and enables individuals to access information more simply and efficiently, 
as “the proactivity of the environment lightens the cognitive load that the user cur- 
rently has to deal with to access information via computers" [4, p. 482]. 

The development of AI is paving the way for a large number of innovative, highly 
responsive, personalized applications and services. However, it is also a significant 
risk driver, particular in the areas of personal security and safety. Artificial intelli- 
gence is a factor in QoL, due in large part to the emotional responses it provokes 


! https://www.pewresearch.org/?s-ambient9620intelligence, visited 1 December 2020, keywords 
for the research engine: AI. 
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(e.g., safety/lack of safety, security/insecurity). Given the threats presented by hack- 
ers and other malicious actors, the protection of sensitive personal data in an envi- 
ronment of widespread AI use is essential and must be an object of constant attention 
for both service providers and the users themselves. Recent scandals indicate the 
kinds of AI security and safety threats that may become increasingly prevalent in 
the future. For example, the cyber-attack launched in May 2017 via the “Wannacry” 
virus infected more than 200,000 computers belonging to individuals in over 150 
countries [5]. Given these risks, the consequences of AI are a major concern for 
individuals that is likely to affect QoL. 

In protecting against threats to private life and overall safety in the context of 
cybertechnologies, individual users must assume primary responsibility. As the 
domain of personal AI technologies continues to grow, individuals must continually 
increase their digital literacy and vigilance against risk. In 2017, in its biannual 
report on information security, the Swiss Confederation stated in an inventory car- 
ried out at the international level that it expected the number of connected objects to 
grow to 20 billion globally in 2030 (compared the 6 billion connected objects that 
existed when the report was issued; [6]). The report also indicated that individuals’ 
vulnerability was “due to the inappropriate safety culture" that predominated in 
information security practices [6, p. 8], including the behaviors of service providers. 
The report demonstrates that cybersecurity ultimately depends on the actions of and 
interactions between several cultures of data protection. The aim should be to define 
appropriate individual and collective practices that would allow the development of 
a “good” safety culture [7]. Developing shared values, norms, and symbols with 
respect to security requires mobilizing not only experts but citizens at large as well, 
the latter of which are the first vectors of risk and have a genuine role to play in 
developing cultures of safety. 

To properly mobilize individual citizens, an analysis of their actual behaviors 
and practices with regards to AI is highly needed as an initial step. Once these cur- 
rent behaviors and practices are understood, developing awareness and vigilance 
should be key areas of focus to allow individuals to ensure their safety and security 
and achieve better QoL outcomes as a result. An individual’s perception of security 
and safety is a central point that must be understood in order to transform behaviors 
and develop prevention practices. The use of foresight processes is a promising 
technique for facilitating such transformation, because it helps individuals and col- 
lectives consider systems of action [8] as a whole (i.e., political, economic, socio- 
cultural, technological, ecological, and legal [PESTEL] megatrends). Additionally, 
foresight processes consider weak signals (i.e., areas where information is incom- 
plete or lacking) and, while encouraging exploration of alternative future scenarios, 
potentially improve participants’ decision-making processes and actions. Rather 
than anticipating foreseeable and unalterable changes and better prepare for them 
(by being pre-active), foresight processes encourage individuals to act in order to 
bring on a desirable future (by being pro-active; [9]). This distinction between the 
(pre-)acceptance of changes (after the fact) and proactive contribution to change is 
important. When proactive actions are identified and perceived by a collective, indi- 
viduals feel more motivated and responsible because they take the future upon 
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themselves instead of simply dealing with the changes. In addition, in the case of 
AI, individuals are likely to be more involved in developing preventive actions to 
the extent that they feel safe and secure in their own use of AI. 

Considering the need for research regarding AI use and attitudes towards AI 
among individuals, as well as the potential for collective foresight processes as a 
tool to drive individuals’ future actions on AI, this chapter seeks to answer the fol- 
lowing questions: What attitudes and motivations characterize individuals’ current 
AI use, and what behavior and practices of AI use (or non-use) do they engage in? 
What future scenarios for AI are possible in Switzerland? How will companies, 
administrations, and the public sector in Switzerland be affected by developments 
in AI? How can individuals in Switzerland seize opportunities to improve security 
and safety and respond to AI-related threats? To answer these questions, we carried 
out a study between September 2016 and January 2017 involving 1000 university 
students in Western Switzerland. This chapter reports the results of the study and 
discusses the implications of these results for efforts to ensure individual users' 
Cyber-security and safety in the midst of technological developments and improve 
QoL in the long term. 


Methods: Futurescaper, Animating Collective Reflection 
and Foresight 


According to some researchers, foresight consists of developing plausible scenarios 
for the future that shed light on imminent decisions and actions [10]. The develop- 
ment of such scenarios facilitates individuals' strategic actions in an uncertain envi- 
ronment [11]. Change is a central concern of foresight, and the scenarios developed 
through foresight explore new opportunities resulting from changes in a systemic 
environment [12]. 

In this study, we used an online platform called “Futurescaper” (Futurescaper. 
com) to encourage forward-looking reflections and operationalize foresight pro- 
cesses. Futurescaper is a collaborative tool that supports the imagining of future 
solutions by a group of participants, each of whom can provide significant contribu- 
tions and insights into specific situations. Tools for foresight processes like 
Futurescaper helps individual users describe and analyze their prospective actions 
in the context of a collectively generated future. Several similar tools also exist, 
such as Scenario Management International (ScMI.de), 4strat (4strat.com), FIBRES 
(fibresonline.com), and the Futures Platform (futuresplatform.com).? Since we had 
the opportunity to use Futurescaper for in-class learning in a course on foresight 
with bachelor's- and master's-level students in Western Switzerland, this chapter 
focuses on this tool in particular. 


*https://rossdawson.com/futurist/futurist-software-and-tools/, visited 1 December 2020. 
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Stakeholders' involvement 


Systemic mapping 
Understanding of consensus views 


Identification of divergent views 
Development of scenarios based on multiple factors 


Fig. 18.1 Contributions of Futurescaper and similar future scenario platforms 


Futurescaper's contributions are qualitative (Fig. 18.1), as it helps groups of indi- 
viduals collectively imagine the change factors influencing potential futures, the 
consequences of these change factors, and the consequences of these consequences. 
Futurescaper facilitates collective reflection by all the individuals within a group, 
allowing participants to propose new combinations of change factors and their cor- 
responding future scenarios. In this study, the new scenarios included opportunities 
to be seized relating to the development of AI and security and safety threats to be 
avoided. Given these opportunities and threats, participants can articulate the atti- 
tudes they should adopt in order to change or pursue the imagined future scenario. 
Participants verbalize courses of action, taking into account change factors that 
other individuals in the group might not have identified. In this way, Futurescaper is 
a future scenario-building tool that encourages proactive action. 

The course on foresight in which the present study was conducted took place 
between September 2016 and January 2017 in Western Switzerland. During this 
course, 1000 bachelor's and master's students participated in a crowdsourced fore- 
sight exercise. The students came from the fields of economics and management, 
engineering, art, and design. A facilitator oversaw the overall process of the exer- 
cise, including configuring the platform and encouraging the participants to brain- 
storm the scenarios with multiple change factors. 

Implementing Futurescaper involves the following four-step process (as shown 
in Table 18.1): project preparation (step 1), stakeholder engagement (step 2), inter- 
pretation and analysis (step 3), and the restitution and presentation of the results 
(step 4). As considerations concerning the general environment (e.g., PESTEL fac- 
tors) are the starting point of any definition of plausible future scenarios, such con- 
siderations are a key factor in the project preparation step [10]. Table 18.1 presents 
a brief description of the actions carried out for each step of the process in the pres- 
ent study, as well as the results of each step. 

To prepare the project (step 1), the facilitator predefined 52 change factors that 
were derived through exploratory research and collection of data concerning AI. The 
facilitator used newspaper articles, scientific and professional studies, blog posts, 
and other documents to provide quality starting content. The facilitator then held 
workshops with students (step 2), each of which began with students viewing an 
introductory video on the current state of AI in Switzerland. In total, 25 workshops 
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Table 18.1 Steps of Futurescaper approach to future scenario building 


S. M. Chapuis 


Step 2: Step 3: Interpretation and 
Step 1: Project Stakeholder analysis (from change factors to | Step 4: Presentation 
preparation engagement future scenarios) of the results 


Desk research 


25 workshops 


Interpretation and analysis of 


An academic paper 


conducted by (average 40 crowdsourced change factors in a book 
researchers students per and future scenario development | Academic article in 
(authors, domain workshop) carried out by researchers a peer-reviewed 
experts) (authors) journal (to be 

52 initial change 2193 337 merged crowdsourced submitted) 

factors (see Annex | crowdsourced change factors 

A) change factors 4 contrasted, plausible scenarios 


along 2 axes 


Table 18.2 Survey questions 


Question 1: What are the change factors that you would intuitively associate with AI? Change 
factors can be political, economic, social, technological, environmental, or legal (PESTEL). 
Answer type: you may select up to 3 pre-registered items or create new ones. 


Question 2: What could the impact of those change factors (answers to question 1) be? Answer 
type: you may select up to 3 pre-registered items or create new ones. 


Question 3: What could the impact of those change factors (answers to question 2) be? Answer 
type: you may select up to 3 pre-registered items or create new ones. 

Question 4: What is the most significant opportunity or threat that you see emerging from those 
change factors (any change factor)? 

Answer type: you may describe that opportunity/threat. 


Question 5: From now on, which actions might you recommend to companies, public 
administrations, and individuals? 


(with an average of 40 students per workshop) were conducted, all of them struc- 
tured the same way, starting with an introduction to AI, then a presentation of the 
survey (see Table 18.2 for survey questions), followed by the students taking the 
survey and a wrap-up. To stimulate students’ initial thoughts and provoke discus- 
sion, the facilitator presented three recent examples of AI in practice (autonomous 
vehicles, smartwatches, and drones) at each workshop and compiled students' reac- 
tions. At each workshop, the facilitator also led a discussion among the students to 
elicit insights regarding their knowledge of AI, their attitudes and motivations 
regarding it, and the behaviors and practices of AI use (or non-use) they engage in. 
The process of engaging the students in step 2 was highly beneficial, leading to the 
identification of 2193 crowdsourced change factors. 

Step 3 of implementing Futurescaper involves interpreting the data collected. In 
this case, the 2193 crowdsourced change factors were reduced to 337 merged 
change factors, which we then analyzed in order to identify interrelationships and 
possible axes along which to consider distinct scenarios. The identification and 
analysis of the merged crowdsourced change factors during step 3 supported the 
generation of four contrasted and plausible future scenarios for AI with respect to 
personal security and safety. We defined two prime axes (see Fig. 18.2) to represent 
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Al for business Al freeze 


Interest in connected objects 


Low 


Fig. 18.2 Possible scenarios for the futures of AI in Switzerland 


these four future scenarios. We then developed 564 proposals directed towards busi- 
nesses, governments, and individuals regarding actions that could be taken to seize 
opportunities and protect against threats related to the use of AI. By outlining this 
study in the present chapter, we carry out step 4, the final step of the process, which 
is to present the results to the international academic community. We hope to also 
present these results in a peer-reviewed article in the near future. 


Results 


In this section, we present the results of the study described above regarding possi- 
ble futures for AI in Switzerland. In the presentation of these future scenarios, we 
consider the implications of future technologies for the security and safety of users 
and the possibility of such technologies leading to improvements to QoL in the long 
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term. As mentioned in the previous section, our crowdsourced foresight approach 
led to the identification of 337 change factors. These allowed for the elaboration of 
the four scenarios discussed below (see Fig. 18.2): “Al for the best,’ "AI down,” "AI 
for business,” and “AI freeze.’ We named each scenario according to the descrip- 
tions provided by the participants of the study. Each of these four scenarios presents 
opportunities related to AI that businesses and governments might seize in the pres- 
ent to improve long-term QoL, as well as security and safety threats that individuals 
can proactively address. As mentioned above, the scenarios were defined along two 
main axes, which were based on two composite change factors whose future out- 
comes were deemed the least certain due to their high degree of dependence on 
social and technological developments. These two change factors are individual 
datafication and interest in connected objects. Datafication consists in moving from 
data to useful information: “insurance companies can, for example, use data relating 
to the movements of the vehicles of their policyholders in order to establish con- 
tracts as close as possible to the risks actually presented by their customers (and no 
longer contracts taking into account their age, their sex, and their driving history)” 
[13, p. 95]. Datafication refers to the quantification of various aspects of our daily 
lives in the form of data and information that can be used for diverse purposes [14]. 
High levels of individual datafication means that large amounts of information are 
extracted from individuals’ data via everyday devices for potential use by compa- 
nies, governments, or the individual themselves. In a society with low levels of 
individual datafication, the amount of data extracted from individuals, or the amount 
of useful information derived regarding individuals’ behaviors, will be lower than it 
is today, either as a result of individuals’ rejection of data-collecting devices or 
because of an improved culture of security around digital devices. High and low 
interest in connected objects/AI are expressed through what we analyzed from indi- 
viduals’ attitudes and motivations regarding AI and the behaviors and practices of 
use they engage in. High and low interest in connected objects/AI refer to the 
degrees to which individuals, businesses, and governments take interest in AI tech- 
nologies and assess the benefits and drawbacks of its various applications. In a 
society with high AI interest, individuals are knowledgeable about developments in 
AI and their implications; in a society with low AI interest, developments in AI may 
be widely used, but there will be little awareness of or interest in AI from the culture 
at large. In what follows, we offer a brief description of the scenarios developed 
based on these and associated change factors. 


“Al for the Best" (A Utopian Scenario) 


This scenario is characterized by strong interest in AI combined with a high level of 
datafication of individuals. In this scenario, individuals view new technologies with 
enthusiasm. They are expert users of connected objects and are eager to exploit their 
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full potential, treating them as instruments to facilitate community-building and 
individual growth. Individuals in this scenario are proactive and willing to share—in 
real time, if possible—the data they generate with the manufacturers of connected 
objects and service providers. They feel safe and secure and are convinced that this 
is the best way to contribute to the continuous improvement of user experiences. 
They are confident and feel free, and they view the impact of AI on their QoL as 
positive. 

In the “Al for the best” scenario, AI becomes a natural extension of the human 
body. At the same time, individuals develop genuine connection with AI (made pos- 
sible by the widespread deployment and generalization of connected objects). 
Artificial intelligence reassures users; it follows and supports individuals’ needs in 
real time. In this scenario, one assumes that there is no possibility of AI harming the 
individual. Secure in their use of AI, individuals are able to use it to maximize their 
QoL in the long term. 

In this scenario, the design of AI technologies is informed by the high level of 
overall datafication of individuals, allowing various highly personalized services to 
be offered to the users. In the “Al for the best" scenario, safety and security experts 
face a surprising question: What is left for them to do? What does society expect 
from them? 


“AI Down” 


The “AI down” scenario is characterized by strong interest in AI combined with a 
low level of individual datafication, where vast data processing capabilities are nev- 
ertheless available to individuals. In this scenario, while connected objects continue 
to accumulate, they do not offer many new opportunities for users nor new security 
and safety risks. The reason for this is that entities and individuals are unable to 
transform the data collected into actionable information or relevant knowledge in 
their particular context. Individuals’ QoL is partially impacted, since interest in AI 
does not result in significant improvements in digital technology or widespread cul- 
tural developments, leading to frustration and a sense of stagnation. 

In the “AI down” scenario, the design of AI technologies is affected by weak 
datafication, and there is a lack of highly personalized services available to users. In 
this scenario, safety and security experts are mainly concerned with the future of AI 
and its potential uses, asking what the potential of AI is and when it will be fully 
realized. It is unlikely that AI affects QoL, since it seems impossible for technologi- 
cal developments to produce changes in culture or daily life. 

Consequently, said security experts must react to new developments in the same 
manner as users to predict as early as possible potential security and safety threats 
and opportunities produced by new services, objects, and classes of connected 
objects. 
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The “AI for business” scenario is characterized by low interest in AI combined with 
high levels of datafication of individuals. In this scenario, individuals have only 
slight interest in AI, although they individually and collectively generate ever- 
increasing amounts of data through their numerous daily transactions. While the 
datasets they produce provide information and high value-added knowledge to busi- 
nesses and government entities, the safety and security of users are also well ensured. 

In the “AI for business” scenario, the data that are produced form a “knowledge 
pool” (i.e., a commons) that is made available to innovators to help them accelerate 
the development of prototypes and shorten the time required for them to reach the 
market. Individuals contribute to open innovation dynamics [15], which most actors 
in the AI field, including computer security experts, have widely embraced. 
Individuals’ QoL is significantly impacted because AI users, whether they approve 
or not, contribute knowledge that leads to innovations in everyday life 
technologies. 

In this scenario, the design of AI technologies benefits from high levels of data- 
fication, but there is a lack of highly personalized services offered to users, as the 
users do not express a need or desire for these services. 

In this version of the future, the large amount of user data available to safety and 
IT security experts allows continuous improvements to data and infrastructure secu- 
rity and greater awareness among experts of the challenges associated with 
cybertechnology. This ensures that businesses do not simply take advantage of indi- 
viduals’ data for their own benefit. Experts are concerned about establishing and 
maintaining high standards of cybersecurity to ensure the safety and security of 
individuals and maximize their QoL in the long term. 


“AI Freeze” 


The fourth and final scenario, “AI freeze,’ is characterized by low interest in AI 
combined with low levels of datafication of individuals. In this scenario, AI consists 
of a variety of gadgets. However, most people are afraid that, sooner or later, a disas- 
ter will occur that compromises their safety and security. There is a lack of confi- 
dence in the manufacturers of connected objects and service providers, which 
ultimately impacts users’ QoL. The latter ask themselves questions such as “When 
is it going to happen?” “Will that connected, autonomous vehicle be hacked and 
cause the death of one or more bystanders?” 

At the same time, people distrust the large companies that collect, analyze, and 
perhaps even resell their data. From the perspective of safety and security experts, 
the “AI freeze” scenario is not a “surveillance” scenario. However, no one questions 
the importance and necessity of the continuous monitoring of data traffic. 
Meanwhile, no one assumes responsibility for detecting attacks and other 
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fraudulent behavior, which can be potentially catastrophic, putting individuals’ 
safety, security, and long-term QoL at risk. 

In this scenario, the design of AI technologies is limited by low levels of datafica- 
tion, and highly personalized services for users are lacking. 


Discussion 


There are numerous consequences of the development of AI for individuals, public 
agencies, companies, and communities. Studies that analyze the implications of AI 
must therefore consider the issue at both the individual and collective levels, includ- 
ing those focusing on AT's impact on QoL. Trends at one level of analysis can have 
chain effects with consequences for others, such as changes in collective employ- 
ment that impact individuals’ health. However, in considering these facets, research 
on AT's social impact should ultimately assess whether AI helps to meet humanity's 
needs or, on the contrary, creates additional problems. To predict AT's future influ- 
ence on a society, researchers must specifically consider the perceptions of the indi- 
viduals in that society regarding AL, since cultures and value systems around the 
world vary widely and are subject to continuous evolution and change. Expectations 
regarding standards of security and safety, for instance, may differ from one popula- 
tion to another. Furthermore, it is important to consider the perceptions of individu- 
als in their particular context, because this ultimately determines individual attitudes 
and drives behaviors. These perceptions, which influence social action, can also 
vary from one individual to another. To predict the potential of AI on QoL in the 
future, individuals’ perceptions of AI must therefore be a key consideration. 

Our study, which surveyed 1000 university students in Western Switzerland 
regarding future AI scenarios in the context of workshops on this topic, increased 
these students’ level of familiarity with AI, allowing them to gain experience think- 
ing about this topic while addressing additional issues, such as personal security and 
safety, that they may not have considered elsewhere. The study also contributed to 
individual datafication, since Futurescaper stores the responses used to develop its 
plausible future scenarios in a large server. Our study corroborates the idea associ- 
ated with the quantified self-movement that practices of ‘measuring yourself’ using 
technological innovations can be highly beneficial to the individual. The quantified 
self-movement, first described in 2007, encourages self-knowledge through the col- 
lection and analysis of data relating to the body and its activities. Recent studies 
[16] have argued that “quantified selfers” are the individuals with the highest levels 
of datafication and interest in connected objects. Unfortunately, it was beyond the 
scope of this study to consider the degree to which participants identify with this 
movement. 

In different research settings, researchers have also defined and evaluated the 
technology of wearables [17]. It would be worthwhile, in future research, to 
imagine or ask participants to imagine future individuals’ attitudes and motiva- 
tions regarding wearables that incorporate AI and their behaviors and practices of 
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use (or non-use). Based on the findings of studies like that of Estrada-Galifianes 
and Wac [17], which analyzed 438 off-the-shelf wearables, new questions along 
the following lines could be introduced to the survey given to students (see 
Table 18.1): What impact could the use of small personal devices have on indi- 
viduals? Could these devices be considered a new organ of the body? Since these 
devices can be worn on one’s head, wrist, legs, torso, arms, and ears, do the quan- 
tified selfers who wear them perceive themselves to be advanced people, cyborgs, 
or humanoids? 

Our study has certain methodological limitations. One of these is the fact that the 
merged crowdsourced change factors used to develop the future scenarios described 
above were derived from only one particular population or category of stakeholders 
[18], namely students from Western Switzerland. Although these students may cor- 
rectly identify future trends related to personal security and safety in connection 
with AI, they are not representative of the entire population of Switzerland. 
Furthermore, the facilitator in this study played a determining role in how students 
engaged in the foresight process. The participants also may have been more moti- 
vated to participate because they knew each other and belonged to the same school. 
One can imagine participants having very different attitudes regarding the future of 
AI in other study contexts (e.g., alone at home or in a country at war such as Syria). 
The results could also be affected by participants’ level of technical mastery or their 
motivation to use an electronic tool like Futurescaper, or scenarios in which partici- 
pants are not allowed or able to use such tools. 

Despite these limitations, our study contributes to research on QoL technologies 
[19] and their future implications. As greater social interaction in environments 
such as the university classroom improves students’ learning and affects future 
behavior, our study demonstrates how QoL technologies like Futurescaper can 
enhance learning by allowing experimentation with new forms of social interactions 
and intelligence that individuals ordinarily would not directly or spontaneously 
engage in, while promoting collective learning rather than solely individual learn- 
ing. By inspiring proactive attitudes towards the future, Futurescaper and similar 
QoL technologies encourage participants to renew their commitments to particular 
courses of actions or, on the contrary, revise them. In the future, QoL technologies 
may be a common good that any individual (or other form of intelligence) in the 
world has the right to freely access and use. 


Concluding Remarks 


In this chapter, we have examined perceptions of personal safety and security in the 
context of today’s cybertechnology— specifically, the development of AI solutions 
and services. We have also discussed the implications of AI in daily life for future 
QoL. In particular, based on the survey responses of 1000 bachelor's- and master's- 
level students in Western Switzerland, we have generated four scenarios for the 
future of AI using Futurescaper, a platform that facilitates foresight processes. 
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Ultimately, these scenarios may serve as a test bed for experts’ future research. We 
have also indicated several strategic options and policies that can improve safety 
and security in the future. Based on the lessons these scenarios present, readers of 
this chapter may adjust their decisions regarding the use of AI or even consider 
alternatives that they had not thought of previously. By representing the potential 
implications of AI in different plausible scenarios, this study also demonstrates the 
need for future research focusing on the ethics of AI use, citizen’s vulnerability to 
Al-related threats, and the relationship between AI and QoL. 

Building plausible forward-looking scenarios and assessing them in relation to 
various strategies or policy options gives governments, businesses, and individuals 
the opportunity, at a relatively low cost, to anticipate the direct and indirect long- 
term impacts of particular decisions on personal security, safety, and QoL. Foresight 
processes like those promoted by Futurescaper can benefit organizations, communi- 
ties, and societies around the world. In the domain of public governance from the 
local to national levels—where decision-making is often plagued by instability, 
uncertainty, and contradiction—the possibility arises of institutionalizing foresight 
practices, which are remarkably inexpensive and accessible in comparison to the 
strategic gains they provide. 
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Appendix 


The list of 52 initial change factors 


Political : Unemployment up 


. Unemployment down 


1 

2 

Economical 3. Information flow up 
4. Telecommuting up 
5. Jobs up 

6. Jobs down 

7. Machine-to-machine interactions up 

8. Logistic performance up 

9. Automation up 

10. Decisions made directly by machines and objects up 
11. Quality of work up 

12. Quality of work down 


Social | 13. Human interactions up 

14. Free time available up 

15. Home support and follow-up for older people 
| 16. Free time available down 
| 17. Decreased human interactions 
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Technological 18. Hyperconnectivity up 


19. Hyperconnectivity down 

20. Remote monitoring on the rise 

21. Geolocation of objects and people up 
22. Comfort up 

23. Rising security 

24. Traceability up 

25. Identification of objects rising 

26. Rising environmental observation 
27. Remote action capability up 

28. Assistance to the human rising 

29. Patents up 

30. Autonomy of machines and objects up 
31. Safely down 

32. Decreased human assistance 

33. Comfort down 


l Ecological 34. Pollution down 


35. Increased traffic flow 

36. Lower energy consumption 
37. Electromagnetic waves up 
38. Pollution rising 

39. Rising energy consumption 
40. Traffic flow down 


Legal 41. Cyberattacks on the rise 


42. Data protection up 

43. Privacy protection on the rise 
44. Information exchange up 
45. Rising safety standards 
46. Data collection up 

47. Falling safety standards 
48. Privacy protection down 
49. Data protection down 
50. Cyberattacks down 

51. Transparency on the rise 
52. Transparency down 
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Chapter 19 A 
A Matter of Distance? A Qualitative Study gss 
of Data-Driven Early Lifestyle Assessment 

in Preventive Healthcare 


Troels Mønsted 


Introduction 


The importance of lifestyle-related diseases (LRD) such as type 2 diabetes (T2DM), 
COPD, cardiovascular disease (CVD) [1, 2] and the consequent importance of 
assessing citizens’ risk for developing LRDs at an early stage is widely acknowl- 
edged, but has proven to be notoriously difficult to achieve. The reasons for this are 
many: Most importantly, health risk behaviors are deeply ingrained in the daily life 
of people or part of valued habits and social interactions, and hence difficult to 
change, as well as many individuals are either unable or unwilling to articulate 
these. In effect, it requires a massive effort to effectively screen a population through 
conventional means such as health checks and interviews performed at regular inter- 
vals by e.g. general practitioners (GPs). Hence, too many citizens remain effectively 
at a distance from preventive healthcare services such as smoke cessations courses, 
dietary consultancy, physical exercise or even sleep. 

From a public health perspective, decreasing the distance between citizens and 
preventive care services is essential in the sense that such health services must be 
available, accessible, affordable, and acceptable for the citizens. Currently, LRDs 
such as T2DM, asthma, COPD, CVD, and certain types of cancers account for 
50-60% of all hospital admissions [3], and have a significant impact on the life 
expectancy and quality of life of the individual by affecting both the physical well- 
being and mental health [4-7]. LRDs are often, although not exclusively, incurred 
by health-risk behaviors such as smoking, harmful use of alcohol, poor diet, and a 
sedentary lifestyle [1, 2], and it is predicted that it will lead to a surge in LRDs if 
rates of obesity and physical inactivity continue to increase [8, 9]. 

Health-risk behaviors, early symptoms, and socio-demographic conditions can 
in principle be used for identification of individuals at risk [10-12]. This requires a 
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proactive approach in which providers reach out to people at risk while their condi- 
tion is still revocable, rather than merely react and provide treatment when people 
eventually reach providers and get diagnosed because the damage is beyond 
repair [13]. 

Current initiatives in preventive healthcare in developed countries have primarily 
focused on improving the ability of healthcare services to take responsibility by 
reaching out to citizens, for instance through information campaigns and systematic 
health checks performed by general practitioners (GPs) [14, 15]. Experience, how- 
ever, shows that these approaches are either ineffective or very resource intensive 
for the healthcare systems. 

While the preventive healthcare system struggles to reach out to citizens, recent 
trends suggest that digital technologies hold great potential for enabling people to 
assess and monitor their own health condition autonomously or in collaboration 
with health professionals. Under the umbrella term quantified-self, this entails vari- 
ous types of devices enabling production of data representations of the behavioral 
patterns and health condition of an individual, which in turn can support health 
management and lifestyle change, leading to better life quality in the long term. The 
purpose of this chapter is to investigate how quantified-self technologies can con- 
tribute to decreasing the distance between preventive healthcare services and citi- 
zens by addressing the following two research questions: RQ1: In what ways are 
citizens at a distance from the preventive healthcare system? RQ2: And how can 
quantified-self technologies enable citizens to decrease this distance? 

Based on a qualitative case study of the pilot implementation of a preventive 
healthcare intervention in Denmark, this chapter first analyses the challenges faced 
by citizens for engaging in early assessment of risk for LRD and then proceeds to 
discuss the potential of quantified-self technologies to increase the accessibility to 
preventive healthcare. 


Background 


Towards Data-Driven Approaches to Early Detection 


In Denmark, the responsibility for supporting citizens (5.8 million) through lifestyle 
change is shared among municipal health centers and GPs. These health services are 
tax funded, free to use for all citizens, and developed to support citizens through all 
stages of lifestyle change. However, while citizens can get referred to lifestyle 
offers, for instance by their GP, it is their own responsibility to reach out in case they 
want support. The municipalities (98) are responsible for offering consultancy and 
support for lifestyle change through services such as smoke cessation courses and 
dietary consulting. The GPs (app. 3500) are responsible for identifying citizens with 
risk behaviors and for monitoring their health status over time. Preventive health- 
care first and foremost involves assisting citizens in converting to a healthier 
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lifestyle, and in retaining this for a lasting period of time [16-20]. Conversion not 
only involves supporting behavior change through, for instance, improved diet, 
physical exercise, or smoking cessation [11], but also generates motivation in the 
citizens for engaging in behavior change, and ensure that the preventive care offers 
match the person’s stage of readiness. Likewise, retention is challenging as health- 
risk behaviors are often deeply ingrained habits and preventive care must therefore 
extend over sufficient time to allow new habits to form [21—23]. Without effective 
assessment of the lifestyle of citizens at risk of developing LRD, and without the 
means to attract these to the right lifestyle offers, preventive healthcare will, how- 
ever, remain at a distance to the citizens. 

Recently, the manner of which analysis of big data in healthcare can assist early 
detection by enabling analysis of more comprehensive, diverse, and timely health 
data has attracted significant interest. This trend stems from the fact that a substan- 
tial amount of relevant data is already accumulated in, for instance, electronic medi- 
cal records, as well as quantified-self technologies which facilitate the production of 
new data representations of the citizens. Hence, relevant data can originate from a 
combination of sources, including clinically-based electronic medical records 
(EMRs) and other health information systems used to routinely collect patients’ 
data, as well as from sensors and mobile phone apps, and crowdsourced information 
[24]. In addition, advancements within artificial intelligence and machine learning 
provide new opportunities for more efficient analysis of big data and for seeing new 
patterns in these data [25]. This approach, however, comes with its own set of chal- 
lenges. First of all, it requires a digital infrastructure that is capable of handling the 
high volume, velocity, and variety of data [25] and often extracting data from other- 
wise non-integrated and incompatible health information systems [24]. Furthermore, 
it requires that the clinical models, workflows and organizational procedures for 
offering preventive care are adjusted to accommodate this potentially more precise 
way of detecting risk in a larger population of citizens. Lastly, this trend may raise 
concern about the risk of mental stress incurred by false positives produced by 
information systems, privacy of citizens and attention towards the potential mental 
response that increased awareness of risk may bring [24]. 

Even more importantly, this may not solve the fundamental challenge of bringing 
citizens closer to the preventive healthcare services, as conventional data-driven 
approaches assume an interest and willingness from the citizens to participate. One 
example is the Danish research and development initiative, TOF—Danish abbrevia- 
tion of ‘Early Detection and Prevention'.! The purpose of this project was to develop 
an intervention that enabled detection of health-risk behaviors in the broad popula- 
tion of citizens, and in particular among those not in frequent contact with the 
healthcare system. The core of the intervention was an information system that 
automatically stratified citizens into risk groups in order to connect them to a suited 
lifestyle offer either in municipal healthcare or general practice. 


! The details of the TOF intervention will be introduced in the findings. 
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The TOF intervention aimed at ensuring ‘right to health’ [26] by (a) securing 
availability of prevention care services for all citizens, (b) creating accessibility by 
inviting citizens through eBoks (a secure, public electronic mailbox), (c) economic 
affordability, as participation was free for all citizens, (d), acceptability, as the inter- 
vention was non-discriminatory and inclusive for all citizens, (e) and quality, by 
basing the intervention on existing, well-established care services. 

The TOF intervention has been under development since 2009, and in 2012 the 
clinical precision of the stratification algorithm was confirmed through a feasibility 
study [12]. In addition, the intervention was tested through two full-scale pilot 
implementation projects in 2016 and 2019, each lasting 3 months, and including a 
total of 4201 citizens (2016: N = 2661; 2019: N = 1540) [27, 28]. The research 
evaluations of the pilot implementation projects proved the ability of the TOF inter- 
vention to attract a large number of citizens and facilitate automatic stratification 
based on citizen-reported health information. However, the research evaluations 
also highlighted that while this type of data-driven approach could in fact identify 
citizen in moderate to high risk of developing LRD, it mainly succeeded in retaining 
those with fairly moderate health risk behaviors and with a relatively high level of 
health literacy [28]. Hence, there is a strong indication that this type of data-driven 
approach to detection of citizens in need of preventive healthcare alone cannot 
decrease the distance between preventive healthcare services and the broad popula- 
tion of citizens, and especially those who may need it the most. 


Quantified Self 


An issue of existing data-driven approaches to early detection, such as TOF, may be 
that they maintain a traditional division of responsibility between the citizen and the 
preventive health providers and hereby assume that the distance between citizens 
and healthcare services can be decreased by enabling the healthcare system to just 
reach out to the citizens in improved ways (in the case of TOF via eBoks and the 
software platform ‘the Health Folder’). This alone will, however, not solve the fun- 
damental challenge that citizens must willingly accept and follow up on the offers 
provided in order for disease prevention to be successful in the long term. At the 
other end of the spectrum, a long-standing interest exists in terms of how technolo- 
gies can make citizens engage autonomously in their own health and well-being. It 
is well known that digital technologies can assist people in assessing and monitor- 
ing their own health condition on their own or in collaboration with health profes- 
sionals. Under the umbrella term, quantified-self, this entails individuals engaging 
in self-tracking of biological, physical, behavioral, and environmental information 
using various types of technologies readily available on the consumer market, 
including exercise wristbands and mobile phone applications, as well as equipment 
developed to support specific types of health monitoring [29]. Recent numbers from 
the US show that 69% of the adult population track key health indicators such as 
bodyweight, exercise routines or symptoms. While many use pen and paper, digital 
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technologies such as wearables (for instance fitness wristbands or other specialized 
sensor technologies) or smartphone applications are becoming increasingly com- 
mon tools to track physical indicators such as fitness, sleep, nutrition, vital signs, as 
well information mental factors such as mood and stress level, or even interactions 
with one’s social network [30, 31]. In fact, it has been documented that 27% of all 
Internet users in the US of age 18 or older track health data such as body weight and 
exercise routines online, and that 29% of all adults have downloaded health man- 
agement applications [24]. 

Quantified-self technologies are primarily aimed at supporting individual, auton- 
omous health management and intended to make citizens more qualified to identify 
relevant health concerns that require professional assistance and seek healthcare at 
their own initiative. Therefore, they also hold the potential for individuals to track 
indicators of their quality of life and consequently, they are increasingly referred to 
as quality of life technologies [32]. This, however, entails a number of ethical and 
societal concerns related to the movement towards transferring, not only autonomy, 
but also responsibility towards potentially vulnerable individuals [33]. However, 
certain types of care can benefit greatly from using technology to create a stronger 
link between the patient and the health provider. This is not least the case in treat- 
ment of chronic conditions, where telemedicine applications have shown to be valu- 
able, for instance by enabling the patient to take an active role in monitoring his/her 
condition and detecting exacerbations at an earlier stage than in conventional care 
(34, 35]. In contrast to telemedicine, which refer to situations where measurements 
are sent to a healthcare provider as part of a mandated type of care [36], the purpose 
of quantified-self technologies is to enable citizens to monitor health and well-being 
autonomously [32, 36], and hence drive individuals to engage in the pre-diagnostic 
work involved in identifying potential health concerns. This has not yet, however, 
provided the required evidence of efficacy and effectiveness maturity to become a 
medical device and reliably support the citizen in important health decisions [37]. 
Additionally, quantified-self technologies do not by definition bring the citizens any 
closer to the preventive care providers when needed. Hence, the purpose of this 
chapter is to explore how quantified-self technologies can help decrease the distance 
between citizens and care providers in preventive healthcare. 


Methods 


Setting 


The study has been conducted within the preventive healthcare system in the Region 
of Southern Denmark (RSD). RSD is a rural area of Denmark comprised of 22 
municipalities with a total population of approximately 1.2 million citizens. 
Compared to the four other regions in Denmark, the population of RSD has a rela- 
tively low socio-economic standing, and the prevalence of lifestyle-related diseases 
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is relatively high. Like the rest of the country, the responsibility for offering preven- 
tive healthcare services is distributed across two sectors in RSD. First, the munici- 
palities (N = 22) are responsible for offering support for lifestyle change, e.g. 
through dietary consultancy and smoke cessation courses, as well as for reaching 
out and informing the population, for instance through information campaigns. To 
enable this, many municipalities have created physical health centers over the past 
decade where these services are integrated. Second, general practitioners are respon- 
sible for responding to citizens they identify as having significant health-risk behav- 
iors as part of their normal routine in the clinic, and offer support for lifestyle 
change, for instance by examining, advising and monitoring the citizen, or by refer- 
ring the citizen to one of the municipal offers. Currently, these preventive care ser- 
vices are, however, largely uncoordinated, and, hence, it is of high priority for the 
Region of Southern Denmark to increase the cohesiveness of the system. 

The case of the study is the research and development project TOF, which is a 
partnership between a research unit on general practice, the RSD, the Organization 
of General Practitioners, and ten municipalities, and aims to develop and implement 
a data-driven approach to early detection that integrates all preventive care services 
in the region.? The TOF intervention has been under development since 2009. Since 
then, two 3-month pilot implementation projects have been conducted in 2016 and 
2019 involving a total of three municipalities and 4201 citizens [27, 28]. 


Data Collection and Analysis 


The study presented has been conducted as an ethnographic study of the practical 
use of the TOF intervention, which has been followed by the author since 2013. In 
order to identify the challenges for citizens in engaging in preventive healthcare, a 
qualitative field study was conducted during two 3-month pilot implementation 
projects in 2016 and 2019. During each round of data collection, citizens and health 
professionals (GPs, nurses, and municipal health professionals) were interviewed 
about their experiences. All study participants have given informed consent for the 
study. The TOF pilot project, part of which this study was conducted, has been 
approved by the Research & Innovation Organization, University of Southern 
Denmark (SDU RIO). The registration numbers of the approvals are 15/60562 
(2016) and 18/32742 (2019). The main focus has been on the experience of the 26 
citizens included in the study (12 males, 14 females, aged 32-60). The citizens were 
followed during their one to three-month participation in either 2016 (N:13) or 2019 
(N:13). 13 of the interviewees opted out of the TOF intervention before the health 
interview with the GP. 21 of the respondents were interviewed over the phone and 
five interviews were conducted either at the care facilities or in the citizens’ homes. 
Ten respondents were interviewed twice during the pilot implementation in 2016. In 


? Details about the intervention are introduced in the findings. 
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total, 36 interviews (duration of 15-45 min) have been included in this analysis. The 
interview guide revolved around (a) the interviewees' general perception of health 
and previous experiences with lifestyle change, (b) their general experience of par- 
ticipation in TOF, and (c) their experience in using a digital system to prepare and 
assess their own risk profile and connect to relevant preventive healthcare providers. 
All interviews were audio recorded. 

For the purpose of the analysis, the interviews were partially transcribed by the 
author. That is, the author listened through all interviews and transcribed the parts 
of specific relevance to the analysis. In line with the ethnographic approach applied 
in this study, the analysis of the empirical data was developed through three stages 
of writing with the aim of building theory through an inductive and interpretative 
process [38]. This explorative process allows for patterns, occurrences, phenomena 
and theorising to emerge and connections between themes that otherwise seem dis- 
parate and later generalised into a coherent argument [39]. 

The first stage consisted of writing the field notes (notes taken during field visits 
and after interviews). The second stage consisted of writing out data (listening 
through interviews, exploring emerging themes in these and sorting them into cat- 
egories). From this analytical process, the general theme of 'distance' gradually 
emerged as a concept encompassing both the physical and psychological challenges 
involved in connecting citizens to preventive healthcare services through the data- 
driven approach. The third stage consisted of writing up the findings in order to 
capture the multiple facets of distance experienced by the citizens [38]. 


Assessing Risk at a Distance 


A Data-Driven Approach to Creating Risk Awareness 


The fundamental goal of the TOF intervention was to design a data-based workflow 
that would relieve health professionals from the labor-intensive tasks of detection 
and enrolment of citizens, and hereby allow them to focus on supporting the conver- 
sion and retention of improved lifestyles among citizens. To achieve this, the task of 
early detection was distributed to a stratification algorithm design to automatically 
assess the risk profile of potentially all citizens in the region. More specifically, this 
algorithm would assess the risk of developing hypertension, hyperlipidemia, and 
COPD (chronic obstructive pulmonary disease), and, based on this, refer the citizen 
to a relevant preventive healthcare service in either municipal healthcare or general 
practice. To ensure clinical validity, the algorithm was based on four guidelines: The 
Swedish National Guidelines for Disease Prevention [40], COPD-PS screener [41 ], 
the Danish Diabetes Risk Model [42], and the Heartscore BMI score [43]. To enable 
the stratification, two sources of data were needed. The first was pre-existing clini- 
cal data extracted from the GPs' electronic medical records. These consisted of 
structured data, including prescription codes, National Health Service disbursement 
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codes, and International Classification of Primary Care (ICPC-2) codes. The second 
source was citizen-reported information on risk behaviors. To facilitate this, a 
15-item web-based questionnaire was developed covering behaviors such as alcohol 
consumption, smoking, amount of physical exercise, diet (intake of sweets, fruit, 
vegetables, and fish), observable symptoms (shortness of breath, coughing), and 
own experience of general health. 

Hereby, the TOF intervention took a great deal of responsibility for decreasing 
the distance between the preventive healthcare services and the citizens with moder- 
ate to high risk of developing LRD, by providing an intentionally efficient channel 
to convey the required data and the result of the risk assessment. 


Dealing with Risk at a Distance 


A general theme in the interviews was that citizens found the TOF intervention to 
be an interesting approach that facilitated renewed awareness of the potential con- 
sequences of their current lifestyle. While three citizens missed the opportunity to 
report the health information directly to a health professional and get direct feed- 
back, a more general theme was that it was more meaningful for the citizens to 
complete the initial risk assessment in the comfort of their own home. The analysis 
shows that the citizens generally felt reflective and honest when answering the 
questionnaire, although one citizen stated that a health professional would be 
judgmental about the person's way of living, indicating a possible social accept- 
ability bias. Thus, to a certain extent, the format of the TOF intervention resulted 
in some citizens feeling more comfortable with relating to their lifestyle, and illus- 
tratively speaking, TOF brought them closer to the possible risk of this. Yet, the 
quantitative evaluation of the TOF pilot implementation in 2019 showed a rela- 
tively high drop-out rate of citizens. Specifically, 134 out of 358 citizens identified 
with moderate risk completed the health interview with a municipal health profes- 
sional, and 144 out of 321 identified with high risk completed the health check at 
their GP [28]. 

As observed during the pilot implementation of the TOF intervention, the digital 
system was useful for the citizens as a tool for reporting and receiving information. 
Hereby, this system supported the overall intentions of improving the connection 
between citizens and preventive healthcare services by developing an efficient 
method of informing citizens about the potential risks of their lifestyle, assuming 
that this would incite many of them to engage in behavior change, either supported 
by a GP or other healthcare services, or on their own. As expressed by the notion of 
health literacy, this, however, presumes not only the ability of the citizen to obtain 
this information, but also to understand and act upon it [44]. 

The information provided through the automatic stratification did not come as a 
surprise to any of the citizens participating in the study indicating a good pre- 
existing understanding of their health condition, the degree of risk, and the causes 
of this: 
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Iam fully aware of what my issue is and that is smoking. To my knowledge I have no dis- 
eases, so it was not unpleasant for me to participate (in the intervention). (Citizen 5) 


In spite of knowing about the correlation between lifestyle and risk, approximately 
half of the citizens participating in the study did not complete the TOF intervention 
by attending either the health interview with a municipal health professional or the 
health check at their GP. Some citizens altered their lifestyle at their own initiative, 
for instance by introducing moderate changes in their diet or exercise habits (citi- 
zens 18, 20, 22, and 23). Among those who did not change their lifestyle following 
their participation in TOF, the dominant theme emerging from the analysis was that 
they found it difficult to relate to the risk assessment performed by the stratification 
algorithm, or in other words; they felt distanced to their own risk of developing 
LRD rather than to the preventive healthcare services. 

First, the analysis showed that that citizens often found the assessment performed 
by the algorithm to be un-nuanced and not fully reflect their lifestyle the way they 
perceived it, or not respectful of their priorities in life. Part of the reason for this was 
that the questionnaire reflected a very structured clinical logic for assessing health, 
which inhibited their ability to report what they found to be relevant nuances about 
their behaviors. For instance, the questionnaire asked “do you eat sweets every day” 
without specifying the amount. Some citizens who had a small but steady consump- 
tion of, e.g., dark chocolate experienced that they, by answering yes to this question, 
would be placed in the same category as people with a significantly less healthy 
lifestyle, yet faced the dilemma that they had to answer yes in order to respond 
honestly to the questionnaire: 


There were a lot of questions where I felt that I couldn't be precise in my answers, so I had 
to choose the lesser of two evils. (Citizen 2). 


From a clinical logic, this question was meaningful, as people who respond posi- 
tively to this statistically have a higher risk of developing LRD without considering 
the actual nutritional value of the diet. To the citizens, however, this made it difficult 
to relate to the assessment, in some cases causing them to become frustrated about 
the assessment or even reject it all together. Adding to this, the system was based 
purely on one-way communication, hereby relying on the ability of the citizens to 
ask questions to a health professional at a later stage of the intervention: 


Actually, I was slightly appalled by the answer. Of course, the system should do what it 
does, but the way it was communicated was like, well okay, is it really that bad? With a 
human I could have asked questions about how this conclusion was reached (Citizen 3) 


Second, the long temporal span through which lifestyle may cause LRD obscured 
the importance of changing behavior in the short term. In contrast to acute disease 
that typically manifests itself through symptoms that are immediately present (e.g. 
the flu), the consequence of LRDs appears over a much longer timespan, often years 
or even decades. Coupled with the fact that risk behaviors such as diet or alcohol 
consumption are often deeply ingrained in social habits and cultures and hence also 
contribute positively to the quality of life for many people, a common theme of the 
interviews was that long-term risk had lower priority than short-term benefit and the 
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pleasure, it would be to maintain the current lifestyle. Out of context of the specific 
data-driven format of the TOF intervention, citizens therefore showed a temporal 
distance to the concept of risk, which formed a barrier for acting upon this 
knowledge. 

Certain factors, however, provided a push towards engaging in lifestyle changes. 
Most notably, it created awareness when citizens looked at themselves and consid- 
ered the consequences of LRDs in the light of someone they held dear, for instance 
family or friends, who suffered or had passed away due to avoidable disease: 


It made me look at myself in a new perspective. There were questions (in the TOF question- 
naire) about my social network and family. I realized that there have been heart conditions 
on my father's side. Compared to his age when he developed his heart disease, that will be 
me in 7 years. So perhaps I should start to eat healthier (Citizen 12). 


An even more significant trigger was the prospect of losing the ability to fulfill their 
moral social responsibility, for instance by being a good parent, spouse or friend in 
the years ahead. However, this was not directly supported by the TOF intervention 
and, hence, it remained a challenge for the citizens in the study to make long-term 
risk a present, short-term concern. 


Summary of Findings 


In response to RQ1 “In what ways are citizens at a distance from the preventive 
healthcare system?" , this study has found that citizens can both be physically and 
psychologically distanced from preventive healthcare services. While the physical 
distance did not impose a barrier for any of the citizens participating in this study, 
the psychological distance was prominent. More specifically, these citizens felt dis- 
tanced to their own risk of developing LRD. This was either because they did not 
recognize themselves in the assessment performed by the algorithm, or because the 
concept of risk to many was difficult to grasp because of the long time span until it 
affects one's health and well-being. 


Decreasing Distance with Quantified Self 


Design Implications 


As argued so far, data-driven approaches to early detection have the potential to 
decrease the burden of the preventive healthcare services when performing stratifi- 
cation of a large population. Furthermore, it creates a space where the citizens in the 
comfort of their own home can engage in their own health without feeling exposed 
to moral scrutiny of health professionals. While such an approach in one sense 
decreases the distance between preventive healthcare services and the citizens by 
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enabling healthcare professionals to actively reach out to a larger number of indi- 
viduals, it does not solve the fundamental challenge of making the concept of risk 
more palpable for the citizens. Rather, as the analysis showed, the citizens often felt 
distanced to their own risk of developing LRD, either because they did not recog- 
nize themselves in the assessment performed by the algorithm, or because the con- 
cept of risk to many was difficult to grasp because of the long time span until it 
affects one’s health and well-being. 

In response to RQ2 “How can quantified-self technologies enable citizens to 
decrease this distance?", two implications can be elicited to guide design of sys- 
tems for data-driven early detection aimed at decreasing the distance between citi- 
zens and healthcare providers in preventive healthcare: 


Design implication 1: The risk assessment provided by the system must be experi- 
enced as relevant by the citizen by accounting for individual variances with 
regard to physical, psychological, and social characteristics. 

Design implication 2: To generate motivation for lifestyle change, a system must 
not only ensure that citizens realize the consequences of the lifestyle for their 
physical well-being and lifespan, but also for their psychological well-being and 
ability to fulfill social roles and moral obligations for family and friends. 


Discussion 


The findings of this study provide insights into what requirements quantified-self 
technologies must fulfill in order to contribute to decreasing distance between pre- 
ventive healthcare services and the citizens in need of these. The most important 
implication is that design should not only focus on creating a connection between 
existing preventive healthcare services and the citizens in order to facilitate efficient 
early detection, but also create fertile ground for citizens to gain an interest in and 
motivation for engaging in the activities related to first detecting potential risk and 
later for converting to a healthier lifestyle. Quantified-self technologies first off hold 
the potential for creating risk profiles that are accurate, relevant, timely and relat- 
able to the citizens. This can be achieved by combining multiple sources of data, for 
instance different types of wearables, such as activity trackers, and self-reported 
behavioral and health information, which is collected over time. While this approach 
may require technical skills and interest from the citizens as well as alignment of the 
underlying risk assessment model with clinical guidelines, it will address the most 
important issue related to existing data-driven detection models, namely that citi- 
zens gain influence on shaping a representation of their risk profile that takes the 
specific details and conditions of their lifestyle into account. 

Another potential of quantified-self technologies is that they often include a 
social dimension by connecting the user to a community of other users in compa- 
rable situations and life conditions, often for the purpose of creating an incentive to 
perform well, e.g. in relation to physical exercise and weight loss. In the context of 
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preventive healthcare, this can enable the citizens to compare their own lifestyles, 
not only to population statistics, but also to how different lifestyles affect the abili- 
ties and quality of life of other people, especially other individuals like them (for 
instance with regard to age, gender, socio-demographics, family context, etc.). 
While this approach requires conscientious considerations towards the potential 
ethical and privacy-related issues that it may entail to induce citizens to expose very 
private information about their lifestyle to others, it will address an important bar- 
rier for engaging in early detection and lifestyle change, that is not breached by 
existing data-driven approaches to early health assessment: as found in this study, it 
is very challenging for citizens to relate to the very long-term effects of an unhealthy 
lifestyle. Also, many citizens are not as concerned about how an unhealthy lifestyle 
will affect their own health and wellbeing as they are about how LRDs will influ- 
ence their ability to fulfill their social roles and obligations for people they hold 
dear. Connecting citizens to a social network that can expose them to knowledge 
about how the consequences of an unhealthy lifestyle can also lead to the loss of 
ability to fulfill these obligations is therefore likely to incite more to engage in life- 
style change. 

The study hereby shows that the matter of distance in early health assessment in 
preventive healthcare goes much beyond the challenge of connecting preventive 
healthcare services to relevant citizens, and assisting citizens in reaching out to 
relevant health care services. From the perspective of the citizens, the issue is not as 
much where to go but why to go, and the barrier for creating motivation for this is a 
psychological distance between the citizens and their own risk. 

In a recent study, Wulfovich et al. [37] found that, in order to offer improved sup- 
port for health self-management, quantified-self technologies must be further 
refined by addressing various human aspects of their use. These include the usabil- 
ity of devices and applications, personalization and context-awareness towards the 
user's life conditions, routines, and lifestyle, as well as the lifestyle advise provided 
by the devices must be timely and non-judgmental. In line with this, the results of 
this study suggest that data-driven approaches in preventive healthcare must be fur- 
ther developed to decrease the distance towards the citizens in order to ensure their 
'right to health" [26]. While the physical distance, as demonstrated by the TOF 
intervention, can in fact successfully be bridged by existing digital means, and 
hereby ensure availability and accessibility for citizens, the psychological distance 
remains a greater challenge. From the perspective of the citizens, this first and fore- 
most will require that the early health assessment provided by data-driven preven- 
tive healthcare interventions becomes increasingly nuanced to be immediately 
recognizable and reliable for the citizens and hereby become mentally and cogni- 
tively affordable for the user. Furthermore, the early assessments provided must be 
adaptable to personal differences between citizens to accommodate for the varying 
physiological preconditions as well as circumstances in life that characterize the 
heterogeneous populations to whom such services are offered. Lastly and most fun- 
damentally, the health assessments must take into account that citizens do not only 
experience risk as personal loss of health and capability, but also as the possible loss 
of ability to fulfill social roles and moral responsibilities, in order to produce the 
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intended motivation for lifestyle change. This speaks for, to a greater extent, focus- 
ing on such factors that evidently increase quality of life in the design of such tech- 
nologies, rather than on developing increasingly sophisticated ways to quantify health. 


Conclusion 


As lifestyle-related diseases continue to have a significant impact on the life expec- 
tancy and quality of life of individuals, it is becoming increasingly critical to develop 
efficient approaches to assist citizens through lifestyle change. An essential aspect 
of preventive healthcare is to detect citizens in need of preventive care services at an 
early stage, when the risk is still reversible. Current approaches to early detection 
tend to focus on how the large number of citizens as well as the physical distance 
between these and the preventive care services prevent health professionals in 
screening for risk at a population level. Hence, development of data-driven 
approaches that support citizens in receiving a risk assessment and support for con- 
version of their lifestyle in the comfort of their own home is attracting significant 
interest. Based on a qualitative study of the practical use of data-driven early detec- 
tion in Denmark, this chapter concludes that this approach, while showing great 
promise, does not succeed in decreasing the distance between citizens and preven- 
tive healthcare services. This is that many citizens feel distanced from this type of 
risk assessment due to a lack of personalization as well as they find it difficult to 
relate to how an unhealthy lifestyle will affect their health and quality of life in the 
long term. Quantified-self technologies may help address these barriers by provid- 
ing an opportunity for producing a more personalized representation of users’ life- 
styles and helping them gain autonomy and influence in creating life changes and a 
better understanding of how their quality of life is actually affected by an unhealthy 
lifestyle and what are actual behavior changes that would fit their social context. 
This will entail the psychological distance between citizens and preventive health- 
care being shortened from a citizen perspective, rather than a system perspective, 
leveraging the human intrinsic motivation. 
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Chapter 20 (f 
Assessment of Activities of Daily Living iim 
Via a Smart Home Environment 


Qing Zhang and Mohan Karunanithi 


Introduction 


Globally, the population is aging. It is expected that by 2050, 1 in 6 people will be 
over the age of 65. It is a significant increase from the current rate of 1 in 11 [1]. In 
Australia, the population aged 65-and-over is 1 in 7, and it is expected to rise to 1 in 
5 by 2050 [2]. This increase in the aging population, accompanied by the prevalence 
of age-related chronic diseases, presents significant challenges to public healthcare 
policymakers, societies at large, and individuals themselves [3]. An increasing 
shortage of care staff exacerbates it due to the aged care and nursing industry's 
inability to attract the workforce and the departure of existing staff from the work- 
place as they get older themselves. As a result, this will not only reduce access to the 
already limited residential care placements, but families will subsequently be 
expected to meet the health, social, safety, and other daily needs of their older par- 
ents. Hence, the future of aged care will increasingly move to the community or 
home [4]. As advances in technology and the Internet of Things (IoT) are becoming 
rapid and part of everyday life, it is inevitable that smart home developments will 
pave the way for future technologies supporting older people in the community and 
enable their quality of life as they age. 

To ensure the health and well-being of older people in the community or home, 
an accurate assessment of the functional independence measures is crucial. Activities 
of Daily Living (ADL) is a good predictor of a wide range of health-related behav- 
iors [5]. Hence, it is used routinely in the clinical setting and before hospital dis- 
charge, and patients return to home. Over the past 40 years, more than 43 indexes of 
ADL have been published to determine the fundamental functional disability status 
of both patients and the general, undiagnosed population [5]. The Katz ADL scale 
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is arguably the first such instrument used in the clinical framework [6]. The Katz 
ADL assessment requires evaluating activities pertaining to bathing, dressing, toi- 
leting, transferring, continence and feeding. This instrument scores each activity 
with 1 if an older person can achieve it independently and 0 if it is dependent on 
assistance. Hence, the Katz ADL index scores will range from 0 to 6, indicating an 
older person’s ability to function as being dependent (0) to independent (6), respec- 
tively. Barthel’s ADL score [7] is very similar to Katz’s ADL score. It is used to 
measure the performance of ten daily living activities, including feeding, bathing, 
grooming, dressing, bowel control, bladder control, toilet use (hygiene), transfers, 
mobility, and use of stairs. The possible scores range from 0 to 20, with lower scores 
indicating increased disability. There are also other ADL scales to measure more 
sophisticated functional independence, such as the full range of activities necessary 
for independent living in the community [8], for stroke patients receiving in-patient 
rehabilitation [9], and for patients with cognitive impairment such as Alzheimer’s 
disease or dementia [10]. 

The challenge is that ADL assessment approaches in the clinical setting are still 
based on subjective assessments from a clinical assessor [11]. There are several 
drawbacks to this approach. First, it is neither representative of the individual older 
people’s living environment nor reflective of their daily living routines. Second, it is 
a subjective measurement because it relies on a clinical staff’s observation and cor- 
responding responses from the individual older people being assessed. Hence, this 
approach’s reliability is neither objective nor consistent for it to be reliable [12]. 
There is also evidence of varied ADL-based assessments, which are likely to be 
skewed by different individual responses to questions. These responses could vary 
from routine- or memory-based answers to socially desirable ones, which could be 
different for each individual depending on their culture, language, and educational 
backgrounds [13]. Furthermore, communication barriers from those with cognitive 
impairment could have significant implications on achieving reliable ADL assess- 
ment. Thirdly, current assessments are clinically resource-intensive, particularly 
when applied in a home setting, making them impractical for long-term care of the 
older people or disabled populations. 

To address subjective bias and reduce human resource time on home-based func- 
tional assessments, the “Smart Home’ concept was proposed in the 1980s and 
applied to support independent-living older people’s health and aging [14]. Along 
with the emergence of new technology in mobile computing, smart sensors, and the 
Internet of Things, smart home is becoming topical and relevant in home automa- 
tion and assistance for health and well-being [15—18]. A few smart home-like prod- 
ucts are emerging in the marketplace that employs motion, and movement sensors 
to detect daily activity patterns and the possibility of detecting falls [19, 20]. 
However, the wide adoption and deployment of smart homes in the senior commu- 
nity are still elusive. We believe there are two main reasons which have limited the 
smart home initiatives. First, the cumbersome use of technology stems from the lack 
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of adequate design choices influencing their ease of use by the older community and 
influencing the users’ perception that technology could compromise their privacy 
and security. Second, the lack of utility—in the form of a personalized and objective 
measure of functional independence that determines the individual’s activities in 
their home setting reflects health and well-being. Current functional independence 
measures through subjective assessment, such as Katz ADL and Barthel ADL, are 
population-based one-fits-all models and not personalized to individual profiles. 
Given this state of the art, this chapter presents a design, implementation, and evalu- 
ation of a novel daily activity assessment tool, the objective ADL (OADL) aimed at 
older populations and achieved through fusing data from simple, non-intrusive, 
always-on, wireless sensors placed in a home environment called Smarter Safer 
Home (SSH). 

This chapter is structured as follows. Section “A Smarter Safer Homes Approach” 
describes our SSH platform and the OADL index computed from the SSH plat- 
form’s sensor data. Section “Clinical Trial and Discussion” introduces a small 
observational trial of the SSH platform and demonstrates the OADL index’s effec- 
tiveness. Section “Conclusive Remarks” concludes this chapter. 


A Smarter Safer Homes Approach 


The CSIRO Smarter Safer Home (SSH) platform was designed as a home-based 
passive activity monitoring system without the need for residents’ intervention to 
capture their health state and potential needs for care support and services. The 
platform was designed to be interoperable with commercially available sensors and 
devices. Furthermore, the design included privacy and security considerations, 
ensuring informed consent of all monitoring and data collection and processing 
processes. 


System Requirements and High-Level Architecture 


To enable an appealing and acceptable platform that seamlessly integrates with the 
home of an older person, the SSH platform’s design challenge was to use only non- 
wearable, environmentally build sensors. From the consultation with the network of 
older individuals and their family members, it was clear that wearables could be 
intrusive, burdensome, and introduce anxiety. In general, the criterion for sensor 
inclusion in the SSH platform was for individuals to attend to their normal lifestyle 
while enabling the SSH platform to monitor their behavioral patterns silently and 
derive their activities towards health status assessment. It was also made clear that 
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to maximize the protection of smart home residents' privacy, the SSH platform can 
not use any video/audio sensors. Instead, it should use advanced machine learning 
techniques to analyze raw environmental sensor data and infer, for example, indi- 
vidual's indoor activities of the state. 

The SSH platform was designed as the outcome of multiple iteration cycles, 
takes advantage of wireless sensor technology and health home monitoring to pro- 
vide a smart home integration that aligns with the older individual needs, and 
enables an engagement of informal (e.g., family) support and formal care services. 
SSH employs machine learning techniques that capture an individual's profile of 
Activity of Daily Living from sensor data. Based on that, personal level functional 
independence can be determined. The platform includes a sensor-based in-home 
monitoring system (data collection), a cloud computing server (data analyses), and 
a client module (data presentation) with a tablet app, a family/care-taker portal, and 
a formal caregiver service provider portal. Figure 20.1 shows the architecture of the 
SSH platform. 


Tablet 
Platform 


Service 
Provider 
Portal 


Fig. 20.1 Smarter Safer Homes platform and its three modules for client, family, and care service 
providers 
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The Wireless Sensor Network of SSH 


The SSH platform includes wireless sensors that can monitor the physical environ- 
ment and daily human activities within the home. In each home, environmental 
sensors are deployed with minimal footprints, i.e., they are deployed seamlessly 
‘invisible’ in key areas of the older residents’ home environment. In an observa- 
tional trial conducted in 2013 [21], trial participants reported that they completely 
forgot those sensors 1 week after installation. These sensors communicate in-home 
activity data with a local server through the Zigbee protocol, enabling low-power, 
secure, and reliable data transmission [22]. 

The main types of environmental sensors deployed by SSH include motion, 
power and circuit meter, ambient temperature and humidity, contact and sleep sen- 
sors. Motion sensors detect the presence of people in its vicinity. They are passive 
infrared sensors installed in every room's corner (one motion sensor per room) to 
monitor the location and transition within the home. Power plug sensors and circuit 
meter sensors are either directly plugged into power outlets or connected in the 
meter box to measure home appliances, ovens, and cooktops' electrical power con- 
sumption. Ambient temperature and humidity sensors evaluate the indoor tempera- 
ture and humidity periodically (one temperature and humidity sensor per room); 
usually, they report readings every 5 min. The contact sensor records open/close 
doors (including pantry, fridge, and wardrobe) and windows. Finally, sleep sensors 
are installed under the bed mattress to monitor sleep quality, length, and different 
sleep stages of individual residents. Table 20.1 shows a full list of sensors, and 
Fig. 20.2 illustrates a typical SSH sensor installation in a two-bedroom unit (approx- 
imately 70 m?). Sensors used in the SSH platform are all commercially available 
from Aeotec? [23], Fibaro? [24], and EmFit? [25]. 

All sensors (except the power meter and circuit meter sensors) are running on 
batteries and can last around 8 months in general in a domestic home environment. 
It makes sensor installations flexible and easy because the sensors are untethered 


Table 20.1 SSH sensors details 


Sensors Position Sampling freq. | Value 
Motion Corners in all rooms | When status Binary status 
changes 
Power Power plugs used by | Every minute Kwh 
appliances 
Circuit Meter box Every minute Kwh 
meter 
l Temperature | Corners in all rooms | Every 5 min Degree 
Humidity Corners in all rooms | Every 5 min Percentage 
Contact Doors of room and When status Binary status 
fridge | change | i 
l Sleep Under mattress Every day Duration, efficiency, sleep stages, heart 
rate, respiration rate 
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Fig. 20.2. An example SSH in-home sensor installation (area of around 70 m?) 


and can be positioned less intrusively and close to the activity being assessed, inde- 
pendent of a power source. It also benefits easy sensor maintenance. Furthermore, 
sensor communication generally requires little bandwidth and is relatively insensi- 
tive to latency. Energy-efficient communication protocols and event-based commu- 
nication strategies can be applied, i.e., only uploading sensor data whenever an 
event change has been detected. In this way, the sensor's battery life can be greatly 
extended. 


Activity Recognition from Ambient Sensors 


Raw sensor data collected at indoor environments are first uploaded to the CSIRO 
secure cloud sever, to be initially processed to extract activities of daily living. 
Figure 20.3 illustrates an example of data samples collected at home by the SSH 
platform, where the ID represents a unique sensor ID (Table 20.2). 

From the raw sensor data, activities are recognized by applying machine 
learning and pattern recognition algorithms. Specifically, the SSH platform eval- 
uated five types of ADL, including Mobility, Hygiene, Dressing, Postural 
Transferring, and Meal preparation, which are included in many ADL instru- 
ments such as Katz ADL Barthel ADL [6, 7]. Specifically, Barthel ADL assess- 
ments' mappings compared to the types of ADLs evaluated by the SSH platform 
are listed in Table 20.3. 
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Daily mobility score of an independent living resident over 9 weeks 
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Fig. 20.3 An example of motion sensor data and mobility scores of one clinical trial participant. 
(a) One day’s motion sensor data in a home (b) Daily mobility score of an indepenent living 
resident over 9 weeks 


Table 20.2 Examples of sensor data collected in-home 


ID Type Location Timestamp Value 
17 Motion sensor Bathroom 2016-08-14T07:24:00 1 

31 Power sensor Kettle 2016-08-14T08:00:00 0.9 kw 
18 Humidity sensor Bedroom 2016-08-14T08:10:00 58% 
19 Temperature sensor Bedroom 2016-08-14T08:22:00 25.6 °C 
23 Contact sensor Front door 2016-08-14T08:15:00 1 


Table 20.3 Mapping Barthel ADL to SSH ADL 


Barthel ADL SSH ADL 
Mobility/stairs Mobility 
Bowels/bladder/grooming/toilet use/bathing Hygiene 

Dressing Dressing 

Transfers Postural transferring 
Eating Meal preparation 


It is important to note that the Barthel ADL instrument evaluates whether an 
older individual can eat meals independently. However, due to limitations of tech- 
nology and privacy violation concerns, we only monitored meal preparation instead, 
assuming the meal consumption was the natural next step. Table 20.4 lists types of 
ADL recognized by SSH, the assessment criteria, and the connected sensors. Every 
day for each recognized activity, we compute its own objective ADL scores, includ- 
ing S_mobility, S_hygiene, S_dressing, S_transferring, and S_meal, representing 
the individual’s daily performance in a given area. To compute an objective ADL 
score, we utilize the assessment criteria for a given domain (e.g., hygiene) and com- 
pare it against normal performance, i.e., performance during the baseline period, 
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Table 20.4 ADLs monitored by the SSH platform 


ADL Assessment criteria Sensors implied 

Mobility (S mobility) Indoor movements Motion in all rooms 

Hygiene (S pyciene) Grooming, showering Bathroom motion, humidity, and 

temperature 

Dressing (S dressing) Access wardrobe/cloth | Wardrobe motion and contact 
room 

Postural transferring Sit/stand from chairs/ Motion in all rooms 

(S. transferring) beds 

Meal preparation (S meal) Cooking, accessing Kitchen motion, contact, power, and 
fridge circuit 


defined as 28 days after the SSH platform installation. Specifically, any objective 
ADL score ranging from 0 to 100 represents the likelihood of the assessed activity 
to be normal, with 0 meaning absolutely abnormal, i.e., very low, and unlikely activ- 
ity performance such as no mobility for a whole day; and 100 meaning normal daily 
activity. In the subsections below, we explain the calculation of individual objective 
scores and the overall objective ADL (OADL) score. 


Mobility, Dressing, Postural Transferring (S mobility, S dressing r S—transferring) 


These three ADLs are mainly evaluated through motion and contact sensors. From 
motion sensors deployed in all rooms, the in-home motion patterns can be inferred. 
With details from contact sensors, SSH can recognize the indoor movement activity 
and compute its performance scores, i.e., S mobility. Figure 20.3a shows an exam- 
ple of motion sensor data from different rooms in one participant's single day. 
Figure 20.3b shows 9 weeks mobility scores within a home when the home resident 
was in a rehabilitation period after hip replacement of the same participant. We can 
see some of the days in this 9-week, the participant's mobility performance is abnor- 
mal, i.e., performance scores are less than 50, which is much lower when compared 
to the performance score during the baseline period participant. 


Hygiene (S hygiene) 


This activity represents how well the resident maintains hygiene status, inferred 
through bathroom access and water usage. Hygiene-related activities can usually be 
recognized through significant changes in the humidity sensor readings in the bath- 
room. As illustrated in Fig. 20.4a, around 19:00, SSH assessed a shower activity 
when applying peak detection techniques in humidity sensor time-series data. 
Similarly, the hygiene scores can be calculated from humidity readings and bath- 
room access occurrences, as illustrated in Fig. 20.4b. It is interesting to notice that 
the hygiene was a bit low from September to October when the weather was cold 
but was higher towards the end of October when the weather became warmer in the 
trial location. 
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Meal Preparation (S_meal) 


Extracting meal activity implies the use of multiple types of sensors deployed in the 
kitchen area. We assume that when all kitchen sensors report data within a short 
period, meal preparation is happening with high probability during that period. 
Specifically, the SSH platform applies multi-scale pattern recognition and unsuper- 
vised learning techniques to evaluate the probability of a meal preparation activity 
and the intensity of kitchen appliances usages as its corresponding score. 
Figure 20.5a shows one day’s kitchen sensor events. SSH computed a meal prepara- 
tion between 11:07 and 12:55 with a 98% confidence and a meal preparation 
between 20:01 and 22:10 with 30% confidence. Figure 20.5b shows daily meal 
preparation probability scores over 9 weeks. 
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Fig. 20.4 An example of humidity sensor data and hygiene scores of a clinical trial participant 
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Kitchen sensor data. SSH computed a possible lunch preparation 11:07 ~ 12:55. 
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Daily meal preparation score of an independent living resident over 9 weeks. 


Fig. 20.5 An example of kitchen sensors data and meal preparation scores of a clinical trial 
participant 
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Computing Objective ADL in a Smart Home (OADL) 


Having the daily ADL value S_mopitin» S_nygienes S—dressings S—transferrings ANA Smear COM- 
puted leveraging all the environmental sensors in a smart home, the objective ADL 
score, i.e., OADL, can be calculated through their aggregation. The OADL repre- 
sents the total health and well-being status of home residents in a day: 


OADL = F ( S — mobility + S —hygiene + S —dressing + S —transferring T S — meal ) 


Like the individual ADL scores (S 7), the OADL score ranges from 0 to 100, indi- 
cating the likelihood of the OADL being normal, with 0 implying very unlikely and 
100 implying a normal day. Note that S ; represents the activity i, and F is a func- 
tion learned from 28 days baseline data (assuming representative data for normal 
days). In our study, we asked participants to do a self-evaluation of their ADL dur- 
ing the baseline period via Barthel ADL [15]. We then apply Gaussian Process 
Regression to learn F with minimum regression error. 


Clinical Trial and Discussions 


Since 2013, the SSH platform has been deployed and trialed in Armidale, a local 
town of New South Wales in Australia. The study was supported by the Australian 
Centre for Broadband Innovation (ACBI) and CSIRO and recruited participants 
from a cohort of aged care facility residents who live in independent units. The 
study was conducted following Health and Medical Research Human Research 
Ethics (HREC* 12/17). The participants in the study agreed to the installation of 
in-home sensors and data analytics via SSH. The participants consented to 12 months 
of participation in the trial. 

To be eligible to participate in the pilot, participants had to be aged over 70 years 
and have no home care arrangements. Participants with cognitive difficulties were 
excluded. Of those who self-selected (N = 23), 17 signed consent forms; however, 
three residents withdrew before the sensors were installed. Participants’ retention in 
longitudinal trials with the older population was problematic for reasons including 
morbidity, mortality, relocation, or others. Over the course of the 12 months of the 
study (9/2013 to 9/2014), there were further withdrawals. Eventually, we collected 
five complete sets of data (sensor and interview data at 3Overse 5 eligible partici- 
pants were aged between 79 and 88 years (Mean 83.6 + 3.6). There were more 
female (n = 4) than male participants (n = 1). Two participants listed primary or 
secondary school as their highest level of education, 2 had non-university certifi- 
cates or diplomas, and only 1 had a university education. Three interviews of par- 
ticipants’ general daily routine were conducted education level after sensor 
installation [14]. Interviews were recorded and transcribed. Relatives or friends 
were present at most interviews and contributed to the discussion. 
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SSH App 


To access the progress and summarized information derived from the SSH platform, 
participants are provided with a tablet and the preinstalled ‘Smarter Safer Homes’ 
app. The app interface was designed with independent living older people and a 
professional graphic design company during an earlier study [26]. The app displays 
the progress status of their daily activities of living and physical and social activity, 
vital signs (measured through clinical devices). Residents can connect to their fam- 
ily or care services via video conferencing services within the app. An example of 
the app’s dashboard reflecting the daily status of well-being is represented by the 
colored rays (Fig. 20.6). A full-extension, green-ray, indicates the individual’s well- 
being measures are within the expected range for that individual. 

In contrast, two-thirds of an amber ray (not presented within Fig. 20.6) means a 
decline to the unexpected range. Furthermore, a one-third red-ray implies a very 
unexpected well-being measure that should trigger an intervention by the care- 
takers, to clarify the individual’s current well-being. There are five main measure- 
ment modules in the SSH app: Health Check, Sleep, Social, Walking, and Daily 
Activity. The core module—Daily Activity, reflects the individual’s OADL. Besides 
this module, SSH also provides other modules to perform remote physiological data 
measurements, sleep efficiency and sleep stage monitor, social connectivity promo- 
tion, and indoor and outdoor step counting. 
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Fig. 20.6 An example screen for the SSH tablet app 
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Family Portal 


Family members and friends of older people living alone often are anxious about 
their welfare. The family portal allows others to gain an insight into the lives of the 
older resident by communicating some of the information about their everyday lives 
via a website. There are four levels of access that the resident can make available to 
family members or nominated contacts. These levels are No access, Limited, 
Standard, and Full details. Figure 20.7 shows the front page of a family member 
with full access to the smart home data. 


Care Service Provider Portal 


The care service provider portal provides access to the SSH platform for formal 
caregivers, such as aged care service providers, to monitor the participant's profile 
and OADL scores. The Service Provider Portal can present an individual's OADLs 


d 


Fig. 20.7 Family portal available through an internet browser 
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over various periods (weekly or monthly). It can also display OADLs of previous 
days on the dashboard (Fig. 20.8) for trend checking and comparison purposes. The 
portal can be accessed by multidisciplinary healthcare teams engaged in an indi- 
vidual’s care. 


Trial Result 


We measure the similarity between daily OADL and self-reported Barthel ADL of 
each participant by computing their Pearson correlation coefficients. In our pilot 
study, the OADL aligns well with the self-reported ADL with an average Pearson 
coefficient (75%), indicating this novel instrument’s great potential as an effective 
tool for accurate state assessment of the individual. Figure 20.9 shows an example 
of the calculated OALD score and self-reported Barthel ADL from one trial 
participant. 

The results for five trial participants who participated in the 10 months are sum- 
marized in Table 20.5. 


Discussion 
ADLs and Home Environment 


In the SSH platform that we present in this chapter, non-wearable, non-intrusive 
sensors are deployed within the home to monitor older people’s behavior patterns. 
The SSH design choices assume that a modern home’s basic infrastructure is avail- 
able, such as running water, electricity, private toilet, internet connectivity, to facili- 
tate the data collection and processing. The home environment examines the 
principal place where a person lives and contributes to its quality. As shown above, 
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Fig. 20.8 An example screen for the SSH service provider portal 
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Fig. 20.9 Anexample of OADL vs. Self-reported Barthel ADL of a clinical trial participant 


Table 20.5 Similarity between OADL and self-reported ADL of all five trial participants 


Trial participant The similarity between OADL and self-reported ADL 
code Age | Gender | (Pearson Coefficients) (96) 

SCOI 79 |Male | 66 

SC02 81 | Female | 80 

SC03 84 | Female | 58 

SC04 86 | Female | 89 

SCO5 88 | Female | 80 


in Figs. 20.3, 20.4 and 20.5, we can demonstrate a technology-supported assess- 
ment that objectively evaluates its habitant daily behavioral patterns. 

Additional quality measures of the home itself could be derived for a wider 
assessment scope. At first, some existing SSH sensor data such as motion, tempera- 
ture, and humidity could help quantify the quality of different parts of the home 
environment, such as temperature and humidity of the living room vs. bedroom. 
Additionally, the comfort quality of the home environment for the older residents 
could be assessed by deploying additional environmental sensors, such as a light 
sensor for visual comfort, a noise sensor for acoustic comfort, or air quality sensors 
for optimal air quality. Furthermore, given the recent climate change, SSH datasets 
could be leveraged to understand the external temperature changes on the residents’ 
home activities. It has the potential to support the design of new homes for heat 
mitigation in hot areas. 

Additional home environment quality measures, not considered within SSH yet, 
may include self-reported safety or intimacy measures, i.e., whether the residents 
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appreciate the amount of space available for them and if they have opportunities for 
privacy if they do not live alone. Further home environment considerations, like the 
quality of the building’s construction (such as roof leaking and damp), as well as the 
quality of the immediate neighborhood around the home, could also be self-reported. 
However, they are not likely to influence the algorithmic power on the OADL com- 
putation of the SSH itself. The overall assessment of the home environment’s qual- 
ity, including additional metrics mentioned above, can be another future research 
direction of extending SSH towards a generic smart home platform. 


Human Factors Influencing the Quality of Data 


In the first clinical trial that demonstrated the SSH platform’s feasibility for inde- 
pendent living [21], tablets were provided to older participants to observe their 
activity profile through the SSH app towards potentially motivating and encourag- 
ing them for better self-management. Although tablets’ interactive touch screen is a 
natural and friendly communication channel for most people, some participants 
found it somewhat difficult to use, especially when required to tap the screen with 
their fingers. Along with the recent development in voice and facial recognition 
through deep learning techniques, one possible future research direction of the SSH 
platform could include developing and integrating social robots as a virtual assis- 
tant, leveraging voice and gesture recognition as input. 

Although the SSH platform uses only environmental sensors to enable conve- 
nience and reduce the potential intrusion for residents attending to their daily life, 
the major limitation is that this platform is currently reliable only for a single inde- 
pendent person in a single home. This is the case until a reliable environmental 
sensor technology distinguishes multiple residents in the home. Occasionally, this 
requirement could be relaxed for a dual occupancy home because there may be few 
joint activities between the two residents. For example, in our study, we recruited 
two multi-residential homes: one was a mother and an adult child, the other was a 
couple. Indeed, we noticed a decline in the SSH data quality of these two homes. 

Nevertheless, these changes did not drastically affect the OADL score computa- 
tion as the mother and adult child differed in their activities and schedules. Examples 
of this include cooking at different times. In the case of the couple, where one has a 
full-time job and is away from home during the day, the other can be assumed as 
living independently during the daytime. 

Furthermore, to allay individual concern about privacy violation and personal 
information disclosure in a smart home, the SSH platform has strictly been designed 
not to use video/audio sensors to monitor in-home activities. Sensors are located in 
unobtrusive positions, such as ceilings, corners, and so on. Blinking indication 
lights of sensors, especially those deployed in the bedroom, are masked to avoid 
disruption. The SSH platform only records raw sensor data identifiable only by sen- 
sor serial numbers, and no mapping details between the sensors and the home or the 
resident are made available. The individual personal information is stored 
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separately on a dedicated secure server. In our study, most participants forgot that 
the SSH platform was present after 1 week of the installation [21]. 


Smart Home and Quantified Self 


The current SSH platform focuses mainly on evaluating five domains that are 
closely related to the ability of older people to live independently. This platform can 
also be expanded into a self-tracking tool that supports Quantified Self and ulti- 
mately contributes to improving the Quality of Life of individuals [27]. This could 
be achieved by adding additional sensors (e.g., wearable sensors [28]) to the SSH 
platform to extend its capabilities to measure various home and outdoor activities, 
including intensity of physical activity (via accelerometer and heart rate) or stress 
levels (via heart rate). Specifically, the SSH platform could develop a digital profile 
of an individuals' daily activities. By tracking and analyzing changes and trends in 
the digital profiles, the SSH system could facilitate individual's self-tracking, self- 
experimentation, and ultimately self-management, thereby promoting their health 
and well-being. 


Quality of Life Technologies 


The SSH platform is designed to support the independent living of older people. It 
satisfies the definition of a Quality of Life (QoL) technology [29]. It provides an 
object Activity of Daily Living index as a tool to assess the health-related QoL, i.e., 
ADL, of older people. Additionally, it connects older people to their caregivers 
through the family and service provider portals and thus enables timely interven- 
tions when necessary to maintain/prevent the decline of QoL. Furthermore, it also 
provides information to older people to support their goal-oriented self-management, 
in turn facilitating the enhancement of their QoL. 


Conclusive Remarks 


In this chapter, we focused on quantifying the home environment as contributing to 
the individual's life quality and introduced a novel objective activity of daily life 
(ADL) assessment through a smart home platform. This platform enables the aging 
individuals to engage their family/care-takers and/or aged care providers to access 
information about their state in real-time and support services that maintain or 
enhance their health and well-being. Through rich and up-to-date information about 
the resident's progress of functional measures of independence and health status via 
the SSH platform, more informed and timely care and support intervention can be 
provided following individual needs. 
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The novelty of the SSH platform lies in its features of providing an objective and 
personalized measure of ADL components and scoring through a set of miniatur- 
ized non-intrusive and non-wearable sensors in the home environment; and the abil- 
ity to correlate this measure with the self-reported or care-reported status of the 
individual’s health and well-being. The SSH platform allows for dignified aging for 
all older people in the community by facilitating efficient and effective aged care 
delivery. Consequently, the SSH platform can deliver enhancement of the quality of 
life to the older people and peace of mind to their families. 
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Chapter 21 (R) 
Granting Access to Information Is Not M 
Enough: Towards an Integrated Concept 

of Health Information Acquisition 


Maddalena Fiordelli and Nicola Diviani 


Introduction 


The overarching objective of this book is to explore the potential of technology- 
enabled methods and tools for objective, quantitative assessment and improvement 
of Quality of Life. This chapter aims at exploring possible ways to enhance both the 
conceptualization and the measurement of the subdomain of quality of life labeled 
opportunities for acquiring new information and skills. After a brief overview on 
the definition of the subdomain under investigation and its original measurement, 
this chapter will present a summary of current studies aiming at the assessment and/ 
or improvement of the variable, making the point for the urgency to find novel ways 
to conceptualize and measure it. The core of the chapter will be dedicated to the 
discussion of how research around the concept of health literacy, which is concep- 
tually very close to the subdomain of interest and has received major attention 
within the academic community in the last decades, might inform developments 
from the point of view of the contents. On the other hand, we will show how current 
practices in the fields of marketing and computer science could inspire possible 
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advancements as regards measurement. The chapter will conclude with the discus- 
sion of some of the challenges and opportunities for future research on the topic. 


Definition of the Variable “Opportunities for Acquiring New 
Information and Skills” 


The subdomain of quality of life labeled opportunities for acquiring new informa- 
tion and skills has been defined by WHO as “a person’s opportunity and desire to 
learn new skills, acquire new knowledge, and feel in touch with what is going on 
[...] through formal education programs, or through adult education classes or 
through recreational activities, either in groups or alone (e.g. reading)”. The subdo- 
main is included in the environmental domain and refers to the individuals’ feeling 
of being in touch with, and having news of, what is going on around them. The focus 
is on a person’s chances to fulfill a need for information and knowledge, whether 
this refers to knowledge in an educational sense, or to local, national or international 
news, that has some relevance to the person’s quality of life. Depending on one’s 
specific circumstances, this can be interpreted either broadly (e.g., being up-to-date 
with “world news") or in a more limited way (e.g., knowing what is going on in the 
local community). 

The construct is complex, because it comprises both an objective and a subjec- 
tive dimension. The objective dimension refers to the possibility to acquire informa- 
tion in terms, for example, of accessibility of sources of information. These include 
formal education sources, such as the school system, but also informal ones, for 
instance family and friends, which in turn can be accessed through different chan- 
nels and in different formats. The subjective dimension of the subdomain, instead, 
refers to the individual’s ability to satisfy the need of accessing new information and 
developing new skills. 


Current Studies Aiming at the Assessment of the Variable 


The questions included in the original WHOQOL-100 instrument are deemed to 
cover both dimensions of the subdomain. Three questions are used for each dimen- 
sion, as the two are deemed equally important. Questions are phrased in order to be 
able to capture all relevant aspects of acquiring new information and skills ranging 
from world news and local gossip to formal educational programs and vocational 
training. It is assumed that questions will be interpreted by respondents in ways that 
are meaningful and relevant to their position in life [1]. 

Studies observing different population subgroups and cultures used the classical 
WHOQOL-100, WHOQOL-BREF [2] or other widely spread measures of quality 
of life such as the Health-Related Quality of Life score (HRQOL) [3]. Findings 
related to the measurement of acquiring new information and skills are consistent as 
they show a positive correlation between this subdomain and the educational level 
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of the individuals in the sample [4]. Findings are mixed in describing the relation- 
ship between financial resources and opportunities for acquiring new information 
and skills, as also very poor subgroups of the population have a positive perception 
of their environmental quality of life [5]. 

The studies presented used the classical measurement tools, whose psychometric 
properties have been consistently proven across cultures, conditions and against 
other measures [6-8]. A measure of the environmental domain, though, has to keep 
up with the historical changes, therefore, to reflect what the current environment 
actually is in terms of offering opportunities to acquire new information and skills. 
These studies highlight connections with other constructs, and these connections 
point to another very relevant construct that will be presented in the next section. 


Changes in the Information Landscape and the Need to Update 
the Subdomain 


As outlined above, current studies aiming at assessing opportunities for acquiring 
new information and skills still largely rely on the questions included in the 
WHOQOL-BREF. The instrument, however, was developed based on the original 
definition of the subdomain, which dates back to 1994 [9]. In the almost 30 years 
after the development of the instrument, however, a major societal change has 
occurred: the advent and the global diffusion of the Internet and affordable personal 
Internet-enabled technologies and its consequences. The magnitude of this change, 
moreover, makes it something that cannot be neglected by researchers interested in 
studying this phenomenon and urges them to reflect on possible ways to update both 
the conceptualization and the measurement of the subdomain to better reflect 
today's reality. First, the Internet has allowed people worldwide to have access to an 
unprecedented number of sources of health-related information on virtually every 
possible topic [10]. Second, the possibility offered to everyone by the new media, 
independently from background or qualifications, to contribute to the discussion 
online, has contributed to the *mushrooming" of websites, blogs and social media 
posts providing unverified information of varying quality [11]. 

How does this societal change affect the subdomain opportunities for acquiring 
new information and skills? On the one hand, it makes the issue of access to diverse 
sources and types of information, i.e., the objective dimension of the subdomain, no 
longer a major problem. If, during its infancy, accessing the Web required the avail- 
ability of technologies which were not for everyone, with the relatively recent wide- 
spread diffusion of smartphones information can be truly considered ubiquitous 
[12]. On the other hand, however, the widespread diffusion of online communica- 
tion has created new disparities [13]. One recent concrete example is the declaration 
of Tedros Adhanom Ghebreyesus, Director-General of WHO at a gathering of for- 
eign policy and security experts in Munich, Germany, in mid-February during the 
COVID-19 pandemic. The term “infodemic” refers to an excessive amount of infor- 
mation about a problem that is viewed as being a detriment to its solution [14]. This 
example clearly shows how, over the last years, we have witnessed a shift of the 
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problem from the availability of health related information to the more and more 
essential ability of accessing this information [15]. This is not restricted only to the 
ability to use technology but refers more in general to all the competences needed 
to make good use of the opportunities the technology has to offer to maximize qual- 
ity of life. We argue that in such an information landscape, the subjective dimension 
of our subdomain, namely one’s ability to deal with information collected, should 
become the conceptually predominant dimension, and therefore the most important 
to assess and potentially improve, if necessary. 


Suggested Approach for Quantitative Assessment/ 
Improvement of the Variable 


In order to explore possible ways to improve the conceptualization of the subdo- 
main opportunities for acquiring new information and skills we undertook a critical 
review of the literature in the field of psychology, educational sciences, health com- 
munication, technology, and marketing. This effort serves to explore some con- 
structs that in our view are closely related to the subdomain at stake, by giving 
particular attention to the evolution in the conceptualization and measurement of 
their main dimensions over the last years, but also to innovative ways offered by 
technology to measure them and adapt to them. As regards the conceptualization, 
we will in particular discuss the concept of health literacy and its evolution, also 
because health literacy has a direct link to empowerment and health behavior and, 
in the long term, to improved health outcomes and reduced healthcare costs [16, 
17]. Research undertaken in the field of marketing and technology will instead be 
used as a starting point to suggest possible innovative future directions in 
measurement. 


Learning from Health Literacy Research 


The individuals’ ability to deal with information has been at the center of research 
in the field of health literacy [18]. We therefore believe that the advancements in this 
field could provide precious insights on possible future developments of the subdo- 
main, both from a conceptual and a measurement point of view. The concept of 
health literacy was originally introduced in the 1970s in the context of school edu- 
cation and was initially understood as a set of basic literacy skills (i.e., reading and 
writing) in the health domain [19]. Following the societal changes outlined above, 
researchers in the field started to realize that being health literate entailed more than 
merely being able to access and read health-related information. Already in the 
early 2000s, Nutbeam proposed a new definition of health literacy, which has three 
main dimensions. The first dimension is basic/functional health literacy and entails 
having basic skills in reading and writing to be able to function effectively in every- 
day situations. A second dimension is labeled communicative/interactive health 
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literacy and refers to more advance cognitive and literacy skills, which, together 
with social skills, can be used to participate in everyday activities, to extract infor- 
mation and derive meaning from different forms of communication, and to apply 
new information to changing circumstances. The last dimension, critical literacy, 
entails more advanced cognitive skills that, together with social skills, can be 
applied to critically analyze information, and to use this information to exert greater 
control over life events and situations [20]. From here, also following the growing 
interest in the concept related to the increasing evidence of a link with health out- 
comes [16], among researchers in the fields of medicine, public health, and health 
communication, several authors have contributed to expand the breadth of the con- 
cept. As a result, all the most recent definitions of health literacy recognize the 
multi-faceted nature of the concept and the need to include, besides functional 
skills, the more advanced skills needed to make sense and evaluate the increasingly 
complex information that is available to the public, including media literacy 
skills [21]. 

We believe that the evolution in the concept of health literacy presented above 
could be useful to inform the refinement of the contents of the subdomain opportu- 
nities for acquiring new information and skills and to shift the focus from its func- 
tional dimension to a more communicative and, what is even more important, a 
critical one. Besides the considerable efforts that have been devoted to the concep- 
tualization of health literacy however, a significant amount of scholarly attention 
has also been devoted to the refinement of existing measurement tools and to the 
development of new ones [22]. In the following, we will describe some of the mostly 
used instruments in an evolutionary perspective and briefly discuss the current 
trends and future directions as they have been described in the numerous reviews 
that have been conducted recently both in the field of health literacy. The most com- 
monly used measures of health literacy, to date, are the Rapid Estimate of Adult 
Literacy in Medicine (REALM) [23] and the Test of Functional Health Literacy in 
Adults (TOFHLA) [24]. The first tool measures a patient's ability to pronounce 66 
common medical words and lay terms for body parts and illnesses, while the was 
developed using actual hospital materials and consists of a 50-item reading compre- 
hension and 17-item numerical ability test. Both measures were developed in the 
early years of health literacy research. It has now been recognized by experts in the 
field that these measures do not fully capture the complexity and richness of the 
concept of health literacy, but are limited to its functional dimension, i.e., the ability 
to read and understand health-related information [25]. Starting from this consider- 
ation, many research groups around the world have started to develop new measur- 
ing tools with a broader scope. Examples of such measures are the All Aspects of 
Health Literacy (AAHLS) [26], the European Health Literacy Survey (HLS-EU) 
[27], or the Swiss Health Literacy Survey (HLS-CH) [28]. In contrast with the 
REALM and the TOFHLA, which are commonly considered objective measures as 
they ask individuals to perform a concrete task, the new measures are mostly sub- 
jective. This means that they ask individuals to rate their ability to perform a task. 
Whereas this evolution has substantially improved the content validity of the mea- 
surement, it has been argued that this type of tools do not actually measure actual 
ability but rather confidence or self-efficacy [21]. Moreover, several authors have 
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suggested that new tools need to be developed to overcome the limitations of exist- 
ing health literacy measurement [29]. Overall, despite the advancements in mea- 
surement, tools to assess health literacy are still very traditional and do not seem to 
take advantage, if not in some rare cases, of the possibilities offered by new 
technologies. 


Advancing Measurement Using Insights 
from Marketing Research 


Whereas, from a content perspective, the field of health literacy and its evolution 
might be a suitable example to learn from, it does not seem to provide useful insights 
as regards advancing the measurement of the subdomain under investigation. 

The field of marketing is a perfect example of how it is now possible both to 
acquire precious information about the individuals (e.g., by tracking consumers’ 
behaviors) and to tailor information to their needs, preferences, momentary context 
and abilities. Online Behavioral Advertising (OBA) is also called “online profiling” 
and “behavioral targeting” [30] and its definitions are multiple in the literature. One 
of them is the following from the Federal Trade Commission: ‘the tracking of a 
consumer’s activities online—including the searches the consumer has conducted, 
the web pages visited, and the content viewed—in order to deliver advertising tar- 
geted to the individual consumer’s interest’. This is just one example of the many 
definitions; however, they all have in common two distinguished components: the 
monitoring of users’ online behavior and the use of the monitoring data to target 
future advertising. Behavioral monitoring happens through use of software ele- 
ments called cookies, or simply through the information that we give to our social 
media. In our online activity, everything can be tracked in principle, but also, we are 
giving out much information on specific channels. On the ground of the data col- 
lected the system make predictions of our behaviors and attitude. As a result, we 
receive advertising that is tailored to the research we made, or even in a more subtle 
way, we are exposed to contents because we interacted with a post or we just spent 
more time on it. Because of our actions, be them conscious or not, our network, and 
our history, we are timely tailored with the contents that are more prone to trigger 
an intention or even a behavior of ours. Behavioral data are therefore used to predict 
new behaviors, or even to arouse behavioral change (which usually results in some 
kind of financial gain for service provider). 

This algorithm-driven approach to marketing and advertising is a novelty com- 
pared to the classical “one size fit all" mass media advertising, but also compared 
to the simple targeted advertising made possible by the Internet so far [30-32]. 
Based on a large amount of data, the algorithm can also become more refined, and 
be informed by persuasion and communication techniques, that make our behav- 
ioral change more likely to happen [31]. OBA can simply be based on our online 
activity through the more classical devices such as computers, tablets or smart- 
phones, but it can also be using data derived from wearables and other more sophis- 
ticated devices. Whatever is able to collect and track data about our daily routine, 
our device usage or content consumption, can inform the algorithm for tailoring the 
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content. The ethical and legal considerations about this practice have accompanied 
the development of the field since its infancy. The regulatory frames of data protec- 
tion have been developed worldwide also in consideration of this, and the percep- 
tion of the users towards his data privacy can strongly influence the persuasive 
effect of OBA practice. However, if this practice is disputable because of its ulti- 
mate aims being directed to profit, there is a chance that the mechanism can be 
exploited for higher purposes such as the ones related to the health and the wellbe- 
ing of individuals. 

It was already some years ago when scientists were envisioning technologies 
able to adapt to the health literacy level of an individual [33]. When technologies 
able to improve user knowledge in specific chronic conditions were already a real- 
ity, researchers advocated for intelligent systems able to improve skill deficits in 
health care and basic literacy skills, such as numeracy through coaching. Beyond 
the provision of knowledge, they said, technologies could influence other constructs 
closely related to health literacy, like for instance self-efficacy and motivation for 
behavioral change using persuasion techniques and counseling agents. Information 
technologies could also serve to activate low literate individuals during doctor 
patient encounters by offering a list of questions and issues at hand. Wac's definition 
of Quality of life technologies goes in this direction when describing its aims [34]. 
Technologies able to respond to the needs of the user, and particularly at enhancing 
his/her quality of life are the ones that prove effective in ameliorating health literacy 
and related constructs. 

Despite some first endeavors in this direction, this is not (yet) happening in 
health, at least on a large scale. Mobile health has exploited behavioral assessment 
for content tailoring in specific interventions or for self-management of chronic 
condition [35], but online (neither offline) behavior is not tracked and used in prac- 
tice to deliver a more understandable health content. It would thus be essential to 
follow this line within the health domain. This means to keep developing and 
improving systems that are able to measure needs, preferences, and abilities through 
the individual actions (e.g., measuring health literacy level through Natural 
Language Processing or through real world actions) and to automatically adapt the 
information provided based on this data and the individual's context [36, 37]. 


Open Challenges and Future Directions 


The goal of measuring the entire construct of Quality of Life, the way it is concep- 
tualized by the WHO, is an ambitious one. Every single subdomain of the construct 
would deserve a separate scale covering all its dimensions, and this is true also when 
it comes to "opportunities for acquiring new information and skills". Based on our 
critical review, we conclude that, in the current information landscape, the measure- 
ment of this specific subdomain of the environmental domain (opportunities for 
acquiring new information and skills) should prioritize the subjective component. 
Indeed, individuals must be able not only to access information but also to appraise 
it critically. Only that way the new information and the new skills will contribute to 
enhance quality of life. 
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Health literacy research has shown that taking into account—and working 
towards the improvement of—citizens’ and patients’ ability to critically appraise 
information has several tangible benefits, making it a valuable investment for gov- 
ernments and health institutions. First, it would enable citizens to practice their 
“right to health”, making healthcare services more available and equitable [38]. 
Second, but not less important, it would contribute to the containment of healthcare 
costs, for instance by reducing utilization of non-necessary health services, increas- 
ing participation rates to preventive services, or improving compliance with and 
adherence to treatment plans [16]. 

While developing systems that are able to assess and collect essential data in 
order to adapt information to the individual, we should take into account the ethical 
challenge related to a "tracking" on the one hand, and wrong adaptation effort, on 
the other hand, which would contribute to an exacerbation of disparities. A system 
collecting the wrong measures or interpreting one single measurement as an abso- 
lute indicator would offer information platforms that are too restricted, in terms of 
content, to the "predicted" need and preference of the user. Measurement would 
need to be comprehensive (and valid) not just in terms of constructs and data col- 
lected but also in temporal terms. We need to take a longitudinal perspective in order 
to work on the effective tailoring approach. Beyond that, we can leverage on what 
the Quantified Self movement has supported so far [39]. By getting to know more 
and become more aware about ourselves through technologies, we could contribuite 
to develop a self-determined and an highly democratic process. 


Conclusion 


Our personal digital devices are always with us, are able to track our actions, to col- 
lect contextual information, and even to ask us direct questions. We envision a sys- 
tem able to unobtrusively measure important characteristics of an individual (e.g., 
educational background, emotional state, beliefs, self-efficacy and health literacy 
level, health behaviors in daily life) in the long run together with environmental 
information. This way, we could build an highly tailored system, always at hand, that 
is able to offer information and recommendations that are not just timely but, hope- 
fully, more useful and persuasive, thus effectively and safely contributing to behavior 
change, better health outcomes and the long term Quality of Life of the individuals. 
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Chapter 22 A) 
Using Technology to Predict Leisure gese 
Activities and Quality of Life 


Andrijana Mušura Gabor and Igor Mikloušić 


Measuring Quality of Life (QoL) 


WHO defines QoL as an "individual's perception of their position in life in the con- 
text of the culture and value systems in which they live and in relation to their goals, 
expectations, standards and concerns” [1]. Four areas of life that QoL describes 
includes physical health, psychological state, social relationships and relationship 
with the environment. QoL is often mentioned with related term “subjective well- 
being” (SWB), recently defined as overall evaluation of the quality of a person’s life 
from her or his own perspective [2]. Since concepts that relate to quality of life have 
been defined in numerous ways [3], many instruments and tools are available for 
measuring quality of life. For example, French MAPI Research Institute offers 
access to more than 1000 QoL instruments available through a database [4]. Linton 
et al. [5] did a review of 99 self-report measures for assessing well-being in adults, 
and concluded with a warning about major variability between instruments and the 
need to pay close attention to what is being assessed under the concept of 
"well-being". 

Although many of these instruments show strong psychometric properties, there 
is much debate over self-report as a method to measure quality of life concepts. 
People are, consciously or unconsciously, deceiving themselves or others about 
what really is the truth regarding their own well-being. On the other side, some? 
research show that reaching a decision about someone overall quality of life is not 
based on careful and systematic analysis of all personal experiences, but on emo- 
tionally intensive peak-end moments in a person's life [6]. In contrast to many 
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self-report measures of QoL concepts, there are some measures emerging that 
exclude self-report and rely on patterns of behavior such as intensity of smiling in 
Facebook photos [7] or the content of tweets posted on Twitter [8]. By analyzing the 
words and topics from collection of tweets, researchers were able to improve accu- 
racy in predicting life satisfaction over and above standard demographic and socio- 
economic controls such as age, income or education. 

This brings us to the trend of “quantified self (QS)", developed due to increased 
availability of wearables and tracking applications. According to Lee [9], QS 
involves extended tracking and analysis of personally relevant data. For example, 
results of Gfk's global study [10] on health and fitness self-tracking reported that 
one in three Internet users track their fitness health via mobile apps or other wear- 
able technology. Many of these self-trackers report that tracking changed how they 
approach to maintaining their health. Some researchers suggest that mobile tech- 
nologies and wearables accompanied by Internet of Things (IoT) will revolutionize 
the way we understand ourselves and live our lives [11]. Besides the value of QS 
movement in learning sciences [9], a line of research that has been recently devel- 
oped proposes a new term—"QoL Technologies (QoLT)", and gives rich insight 
into the ways QS can improve quality of life [12]. QolT collects data from hardware 
and/or software technologies, provides objective and minimally intrusive assess- 
ments of QoL, and via feedback mechanisms aims at improving individual's QoL 
[13]. As such, QoLT gives great promise of providing greater self-awareness and 
opportunities for better quality of life. 


Leisure Engagement and Quality of Life 


WHO defined leisure within QoL model as "Participation in and opportunities for 
recreation/leisure activities as “a person's ability, opportunities and inclination to 
participate in leisure, pastimes and relaxation. The questions include all forms of 
pastimes, relaxation and recreation." [14, 15, p. 66]. Leisure engagement, more 
objectively, is defined by the amount of time, diversity, or frequency of person's 
participation in leisure activities [16]. Defining features of leisure activities is 
intrinsic motivation and freedom to engaging in them, and as such, they predict 
subjective well-being [17, 18]. There are many studies that report on significant 
relationship between engaging in leisure activities and improving quality of life. 
Recent meta-analysis of 37 effect sizes, with more than 11,000 participants, 
reported on strong evidence for the moderately positive association between lei- 
sure engagement and SWB [19]. Moreover, the relationship was mediated by lei- 
sure satisfaction, while measures of the frequency and diversity of leisure 
engagement were more strongly associated with SWB than measures of time spent 
in leisure. 

Usually, leisure activities are classified into two categories: relaxed leisure 
activities (more passive, e.g., sedentary activities) and serious leisure activities 
(more active, e.g., physical activities) [20]. While Passmore [21] talks about three 
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types of leisure activities—active, social, and time-out, Silverstein and Parker [22] 
propose 6 domains: culture-entertainment, productive-personal growth, outdoor- 
physical, recreation-expressive, friendship, and formal-group. Not all leisure 
activities lead to positive outcomes. Studies have shown that relaxed and passive 
leisure activities correlate to less satisfaction and well-being, compared to more 
active or physical leisure activities [23-26]. Kahneman and Kruger [27] offer a list 
of activities with accompanying self-reported measures of positive emotions and 
proportion of time with negative affect. Three activities that lead to most positive 
and least negative affect are intimate relations, socializing after work and relaxing. 
At the bottom of the list, with up to 3096 of time feeling negative emotions, are 
activities of commuting and working. Oishi, Diener and Lucas [28] report that hap- 
piest people are most successful in terms of close relationships and volunteer 
work. On the other side, most commonly chosen leisure activity, watching TV, 
offers only limited enjoyment and needs satisfaction [29]. Overall, it seems that 
physical leisure activities and social leisure time rank high in predicting quality of 
life aspects. 

Results of research on leisure activities and quality of life measures have impor- 
tant implications for developing alternative and less intrusive measures of QoL, 
backed by modern technologies. Although research on this topic is extremely lim- 
ited, there are some research paving the way through the emerging field of QoL 
technologies. 


Quality of Life Technologies 


If we assume that there is a strong relationship between engaging in leisure activi- 
ties and benefits for individual quality of life, especially in the long term, then it is 
also legitimate to assume that alternative measures for leisure activities could point 
to outcomes related with quality of life. This leads us back to quality of life tech- 
nologies (QoLT). Widely available and affordable tools such as personal smart- 
phones or wearable technology provide us with ample of data that can be correlated 
with how individuals spend their leisure activities. For example, smartphones can be 
seen both as tools that provide valuable information about overall activities indi- 
viduals engage in, as well as tools that influence usage of leisure time. 

Because smartphones have become quite pervasive [30], they are associated 
with many different behaviors and behavioral patterns such as social interactions, 
daily activities, and mobility patterns. Smartphone devices are becoming behav- 
ioral data-collection tools, and thanks to their computational power, logs and sen- 
sors, they provide us with unprecedented access to people's social interactions, 
daily activities, and mobility patterns [31, 32]. Research on this topic has only 
started to accumulate. Although there are number of studies that explore the rela- 
tionship between smartphone usage, leisure activities and affective states [33—36], 
they all rely on self-reports. One interesting study, that also used self-reports, asked 
a representative sample of participants to report on the type of communication (text 
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message of voice call) and physical proximity from individuals’ whom the partici- 
pants was in contact with [37]. Authors reported that geographic proximity of indi- 
viduals was related with mobile communication patterns and social leisure 
activities between people who communicate. Let us now image how this study 
would look like if all the data was coming from smartphone devices. If backlog 
communication and geolocation data would be available for participants and their 
contacts from this study, we could determine the type and length of mobile com- 
munication (including recently popular use of messaging apps, beyond standard 
texting or calls), as well as geolocation of, both, caller and receiver. Social leisure 
activity could be, also, identified by geolocation data (e.g. restaurants, movies, 
theaters, concerts, sport games, bars and clubs). Thus, relying only on backlog 
smartphone data, we could learn about communication patterns and make assump- 
tions about quality of life of these individuals, as well. These assumptions could 
take form of key predictors of important life outcomes, such as subjective 
well-being. 

To test whether smartphone data can be predictive of subjective assessments, de 
Masi and Wac [38] used smartphone logger data to predict quality of experience 
assessed by in-situ quality of experience survey. Authors reported that predictive 
model for “good” and “bad” quality of experience can be build using quality of 
service information, mobile application name, user task (e.g., consuming or produc- 
ing content) data within an app and physical activity of user. By combining self- 
reports and quantitative data, authors were able to determine alternative measure of 
quality of experience. Using similar study design, alternative measures for leisure 
activities can be built, as well. For each alternative leisure activity measure, self- 
reported satisfaction and positive/negative affect should be obtained. In Table 22.1, 
we offer a possible list of leisure activities and potential objective indicators, with- 
out affective indication, i.e., if the leisure activity indeed was enjoyable for the 
individual (which would require more physiological assessment with respect to the 
lower levels of stress, more calmness and happiness). 

Having identified some of the objective sources of data for measuring leisure 
activities, we need to mention that future alternative measures of QoL will need to 
include not only the type of activity, but also its frequency, duration and diversity, as 
well as resulting affective state of the individual after the activity is finished (e.g., 
lower stress levels). This calls for different index measures that will have greater 
validity in assessing QoL. Once relevant alternative measures are identified, QoLT 
could help improve quality of life via feedback mechanisms such as notifications, 
reminders or motivational messages, leading the individuals to managing their lei- 
sure activity nest matching their momentary needs and context (e.g., if there are 
resources or opportunities for leisure in location where they are, with whom they 
could meet). With this heightened self-awareness, individuals will be able to change 
their behavioral patterns by overcoming self-regulation barriers that arise due to 
lack of planning or lack of goal progress. This way, individuals will be able to have 
direct impact into their quality of life. 
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Table 22.1 List of leisure activities and potential objective measures 
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Leisure category 


Culture and 
entertainment 


Leisure activity 


Movies/cinema 
(offline) 
Movies (online) 


Type of objective data 


Geolocation data 


Media-service providers 


Source of objective 
data 


Smartphone 


Smart TV, personal 
computer 


Cultural events Geolocation data Smartphone 

Restaurants Geolocation data Smartphone 

Games (offline) Geolocation data Smartphone 

Games (online) Gaming activity, gaming apps Gaming consoles, 
Smartphone 

Apps Smartphone data Smartphone 

TV TV tracking Smart TV 


Internet surfing 


Internet usage data, smartphone data 


Personal computer, 
smartphone, tablet 


Going dancing Geolocation data, wearable data Smartphone, 
wearable 
Personal growth | Books (online) E-book readers Tablet 
Courses (online) | E-mail enrollment notifications Smartphone, 


personal computer 


Outdoor and Walks Geolocation data, blood pressure Smartphone, 
physical and heart rate, steps, stress level smartwatch, 
wearable 
Sports Geolocation data, pressure and heart | Smartphone, 
rate, steps, stress level smartwatch, 
wearable 
Passive Sleeping Blood pressure and heart rate, sleep | Smartwatch, 
apps smartphone 
Relaxing (indoor, | Blood pressure and heart rate, Smartwatch, 
sedentary) movement sensors smartphone 
Socializing Phone call Communication logs Smartphone 
chatting 
Meeting with Geolocation data, communication Smartphone, 
friends logs, calendar data, stress level, wearable, 
talking-to-listening ratio, tone of sociometric badge 
voice 
Internet Internet usage data, social media Smartphone, 
interaction/ activity, messaging apps data personal computer 
chatting 


Club activity 
(church, hobby 
group, etc.) 


Geolocation data 


Smartphone 


516 A. M. Gabor and I. Mikloušić 


Possible Negative Impact of Screen-Time 
on the Quality of Life 


Since 2008, daily hours spent with digital media per adult user have risen from 2.7 h 
to 5.9 h in 2018, according to Meeker [39]. In fact, we rely more on devices for 
leisure time but not all of us (hence it’s important to collect diversity of data to 
quantify leisure) [40, 41]. 

A notable rise in technology use in the American youth from in recent years, 
according to Pew research center (2018) is related to the increase in the smartphone 
ownership. 95% of teens reporting owning a smartphone, with the majority using 
some online social networks such as YouTube, Instagram, Snapchat or Facebook. 
Although the mobile phone technology and instant Internet connectivity is a some- 
what new phenomenon, it has opened up new opportunities for improvements on 
various aspects of our lives and the quality of life. Also, it created new and superior 
ways of tracking and studying human behaviour [31, 42] there are already some 
research point out possible negative aspects of this hyper connectivity. Most of the 
studies done so far have been focusing on possible negative impact information 
communication technologies and social networks such as Facebook have on chil- 
dren and teenagers while some research that linked recently observed decrease in 
well-being and happiness in adolescent populations to the increase in screen activi- 
ties facilitated mostly by widespread use of smartphones [43]. In many ways, focus 
on the negative impacts of new technologies has been a rule in psychological 
research as fears of detrimental impact of television as well as video games spreads 
through the population. For the most part our intuitions on the catastrophic effects 
of for instance video games and their link to violence have for the most part been 
dispelled [44, 45], however it seems that that there is reason for justifiable concern 
regarding this new technological trend. 

As Pew research study suggested, although the participation in social media is 
almost ubiquitous in the teen population, the impact of it is not straightforward [46]. 
About a third of the teens report mostly positive effect of social media such helping 
them connect with friends and family, find information and meet people with whom 
they share interest, as much as a quarter report mostly negative effects of social 
media, with bullying, reducing in person contact, imposing unrealizing views of 
others’ lives and distraction being poised as biggest issues. In a recent review of 
paper dealing with the impact of online social media on adolescent mental health 
Keles, McCrae and Grealish [47] found a positive relation between the use of social 
media and mental health problems. More precisely, the exposure to social media in 
the context of the time spent on the social media networks, type of activity (i.e. 
frequency of checking the profile, number of “selfies”), participant investment and 
addiction to social media were implied to be risk factors for the development of 
depression, anxiety and general psychological distress. High frequency users are, 
especially, under great risk from decreasing their quality of life due to less physical 
activity and poorer physical condition, and having greater connection to their smart- 
phones in spending leisure time [48]. High frequency users report that smartphones 
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make leisure more enjoyable, increase personal freedom, are intrinsically reward- 
ing, make it easier to engage in and experience leisure [49]. 

There are some caveats in these findings. Amongst others, the main issues with 
studies conducted so far include small samples and reliance on self-report measures 
[47]. Also, the majority of these studies are correlational and have a hard time estab- 
lishing the direction of a causal relationship as it could be also likely that people 
with pre-existing psychological disorders spend more time on social networks. 
However, there are some experimental studies that do imply how abstaining from 
social networks can have a positive impact on our well-being. For instance, Tromholt 
[50] conducted an experiment on more than one thousand participants where the 
experimental group was made to abstain from using Facebook for a week. The 
results showed abstaining from Facebook had a positive effect on life satisfaction 
and emotional life, especially for heavy Facebook users and people prone to experi- 
encing envious feelings while using Facebook. Another similar study conducted in 
organizational setting [51] showed that Facebook use and effects happiness in a way 
that it promotes comparison which has negative effect on happiness and that this 
effect was stronger for younger users. Another problem would also be the depth of 
analysis. The quality of interactions that people have on these networks also seems 
to influence the outcomes. In an extensive review of 70 studies dealing with social 
network sites and well-being depression and anxiety showed that the impact of 
social network use marked with positive interactions, social support and connected- 
ness is related to lower levels of both depression and anxiety, while negative interac- 
tions, social comparison and addictive behavior has the opposite effect [52]. Also, 
the positive interactions on social networks might benefit those that otherwise strug- 
gle with face to face communication. 

So far, only a limited number of researchers reflected on the relationship between 
mobile technologies use and leisure activities, drawing from similarly limited num- 
ber studies on the use of mobile phones and their impact on our behavior and emo- 
tions. Lepp [34, 35] sees two major domains of overlap between mobile 
communicating technologies and leisure. First one is facilitation of leisure activities 
through enhancing communication and coordination. Access to this technology can 
help in planning efforts for various outdoor activities but also in creating opportuni- 
ties for spending leisure time in sedentary behaviors for individuals who are less 
prone to face to face interactions. The second is related to the depth of our experi- 
ences whilst engaged in leisure activities. In one hand taking away from our ability 
to isolate ourselves from outside influences and being in the moment, but also 
enhancing our outdoor experiences trough access to information providing naviga- 
tional aid. A creative experiment by Kushlev et al. [53] showed exactly how mobile 
technology can take away meaningful experiences from our everyday activities. 
Participants were asked to navigate through campus and find a particular building 
either with or without using smartphones, and whilst individuals using smartphones 
were more efficient at completing the task the ones that were left to their own 
devices—looking at signs, asking for direction, etc.—felt more socially connected. 
Dwyer, Kushlev and Dunn [54] reported similar findings in a study where partici- 
pants either had their smartphones on the table or put away during a meal. The 
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results pointed toward phone use taking away the enjoyment experienced in real 
world interactions, and in a follow up study the negative effect of smartphone pres- 
ence was found in other types of face-to-face interactions. Mostly, the phone was 
seen as a distraction, preventing people to fully engage with their environments. A 
negative relationship between smartphone addiction and productivity has also been 
reported, with spending time on smartphones taking considerable amount of both 
work and leisure—off time that could have been used on more meaningful pursuits 
[55]. This hyperconnected, interruptive and addiction forming quality of smart- 
phones seems to be a cause of psychological distress. However, there is some proof 
that some of the content available on smartphones, such as health apps aimed at 
promoting health lifestyles and increased physical activity can produce positive 
outcomes (i.e. [56]). 


Conclusive Remarks 


This chapter focuses on the possibilities of combining behavioral patterns collected 
through use of smartphone and other technologies in predicting leisure time and qual- 
ity of life. With the constant rise of internet users and ever-developing technology, the 
abundance of data gives new opportunities for behavioral research that goes beyond 
traditional methods and its deficiencies. One such opportunity includes using behav- 
ioral lifestyle data to recognize leisure activities and outcomes of engaging in such 
activities. Research has shown that life outcomes, such as quality of life and related 
constructs (i.e. SWB), are highly correlated with engaging in leisure activities. 
Throughout this chapter we offered numerous ways of using technology and data 
proxies for assessing leisure activities. We, also, imply that QoLT has a major oppor- 
tunity to impact individual lives through feedback mechanisms that they offer to 
its users. 

There are great advantages to using personal mobile technologies in research on 
leisure activities, and plethora of data it collects is in many ways more reliable than 
standard self-report measures we relied on so far. Especially since there is reason to 
believe that much of our self-report data on our mobile technology use is flawed, as 
recent studies comparing actual smartphone use to self-reports demonstrated people 
grossly underestimate the time spent using our smartphones (i.e. [57]). These trends 
will not only affect individual lives and their quality of life, but will also strengthen 
interdisciplinary research, and possibly transform field of psychology and its 
research methods [42, 58]. 

However, just the extent to which information and communication technologies 
should be fully incorporated in leisure activities is left to be determined. A recent 
interesting step away from digitalization, and toward the trend of so called “Digital 
detox" was reported by MIT Technology review [59]. The so-called Google Paper 
Phone, a product of Google creative lab is basically a piece of paper where a person 
would print out the relevant information for the day—telephone numbers, to do list, 
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shopping list, a map, and use it to go about the day whilst leaving phone at home. 
That proves that maybe QoLT is not the only answer, and the person’s ability, oppor- 
tunities and inclination to participate in leisure, pastimes and relaxation may be 
when technology doesn’t reach. 
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Chapter 23 (f 
The Importance of Smartphone crente; 
Connectivity in Quality of Life 


Alexandre De Masi and Katarzyna Wac 


Introduction 


The World Health Organization (WHO) has defined Quality of Life (QoL) as an 
“individual’s perception of their position in life in the context of the culture and 
value systems in which they live and in relation to their goals, expectations, stan- 
dards, and concerns.” The WHO expands this definition across several domains, 
namely physical and psychological health, social relationships, and the environ- 
ment. In this chapter, we focus on one facet of the environmental domain: the physi- 
cal environment. We explore the availability of mobile network connectivity in 
one’s environment without considering other variables that contribute to this envi- 
ronment, such as noise, pollution, climate, and the general aesthetic. Determining 
the impacts of connectivity on an individual’s QoL is important for considering 
improvements or adverse effects on their day-to-day life. 

Wireless networks have been present in our physical environment since the 
invention of over-the-air transmission of information (ALOHAnet [1]) in 1970. 
Recent developments in communication technology have now made it affordable to 
own a powerful, ubiquitous, network-enabled device. Today, wireless networks are 
present throughout the shared physical environment, especially in the developed 
world and in areas with high population density. Indeed, the accelerated 
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digitalization of the population can be attributed to the global adoption of smart- 
phones. The number of smartphone users reached 3.2 billion worldwide in 2019 and 
will continue to grow [2]. Likewise, the networks that support them have been 
deployed at a similar pace and are continuously updated to cover larger areas and 
upgraded to utilize new technologies (e.g., from 3G to 5G). 

The majority of mobile applications require an Internet connection, and in this 
study we focus on connectivity to mobile networks, whereby human-to-human 
interaction is enabled by computer-based networks. Networks support instant infor- 
mation transfer in various formats, including text, image, and video, and enable the 
necessary interaction between nodes (i.e., people or machines). Furthermore, they 
provide access to a number of services that can be used to improve an individual’s 
decision-making capabilities and ultimately their QoL. A 2018 study by Chan et al. 
[3] found that smartphone use predicts relationship quality and subjective well- 
being, while Kim et al. [4] suggested that the use of information and communication 
technology, such as smartphones, in old age generally plays a positive role in 
enhancing the psychological, mental, and social aspects of one’s QoL. 

This chapter presents features of mobile network connectivity derived from 
smartphone use data collected from different cohorts in the Geneva area (Switzerland) 
between 2015 and 2020. We explore four connectivity features and examine the 
evolution of connectivity during the last 5 years as derived from data gathered unob- 
trusively from the consented mQoL (mobile QoL) Living Lab participants. 

This chapter is structured as follows. We present the literature review in section 
“Related Work”. In section “Mobile Network Connectivity Study: Methods”, we 
provide the study parameters, describe the collected data, and outline the studied 
connectivity features. In section “Mobile Network Connectivity: Results”, we report 
the results obtained from the analysis of the features. In section “Discussion”, we 
discuss the limitations of the study and different approaches to connectivity quanti- 
fication. Finally, in section “Conclusion”, we describe the lessons learned and pro- 
vide recommendations for future areas of work, especially the quantification of the 
impact of mobile connectivity on QoL. 


Related Work 


Mobile Network Connectivity and QoL 


Previous work has shown the benefits of deploying mobile networks in rural and 
developing areas (e.g., Ghana, Nigeria, Kenya, and Tanzania) [5]. Researchers have 
found that it facilitates improved communication between the local population and 
distant services such as health and governance. The same authors have documented 
income growth in the Southeast Asia region in the last 10 years due to the rising 
usage of mobile applications and voice calls as the population gained access to new 
services and information relating to the weather, agriculture, finance, and music, for 
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example. The income growth has only been reported in low-income countries, but 
surprisingly, in 2018, the GSM Association [6] found that the top reason to use 
mobile instant messaging was the same for low, middle, and high-income countries. 
This indicates that the benefits of messaging applications are not the prerogative of 
high-income countries. In recent years, messaging applications have created new 
markets and services that are available on their platforms. For instance, WeChat 
(est. 2011), Facebook Messenger (est. 2011), and WhatsApp (est. 2009) have all 
integrated payment functions into their applications in selected countries including 
China (WeChat Pay), Brazil, and the USA. Before the prevalent use of smartphones, 
the development of mobile payment solutions using a fast and reliable network was 
stagnant. Today, mobile networks are a critical gateway to the digital economy, as 
these solutions have been widely adopted to simplify the exchange of money and 
goods. Overall, 90% of Chinese tourists claim that they would use WeChat Pay 
overseas if given the opportunity [7]. 

The direct impact of broadband network access on GDP per capita has also been 
studied; one investigation found that a 10% increase in broadband penetration has a 
notable impact on GDP per capita, increasing it from 0.9 to 1.5 percentage points on 
average for OECD economies. Furthermore, the authors explained that if digital 
services are established alongside a reliable infrastructure, new services will be cre- 
ated [2]. 

In recent years, the Asia-Pacific region has been improving its environmental 
QoL through connectivity and will continue to do so particularly by way of smart 
city initiatives [8]. Such initiatives are described as cross-sector endeavors that link 
people to public and private infrastructures. Connectivity is crucial for smart city 
services, from the use of Internet of Things devices and a cloud-based platform to 
monitor and analyze air quality at street level, to the publishing of open data by 
public authorities to enable faster development of online-based services. In sum- 
mary, a link between mobile connectivity and QoL around the world has been 
proven to exist—to such an extent that connectivity has a direct impact on a coun- 
try’s GDP. 


Smartphone Apps and Their Impacts on QoL 


The revolution in mobile devices, which have evolved from basic cell phones to 
smartphones, has created a new market for mobile applications. New application 
types were created for those devices, and as of November 2020, the Google Play 
Store hosted 2.56 million different applications across 32 application categories and 
17 game categories [9]. Two application categories that may have a direct impact on 
users’ health are (1) health and fitness, including personal fitness, workout tracking, 
dieting and nutritional tips, health, and safety applications, and (2) medical, includ- 
ing drug and clinical references, calculators, medical journals, news, and handbooks 
for healthcare providers. 
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Health and fitness applications such as food diaries allow users to track their food 
intake for multiple purposes. These applications connect to a central database that 
contains nutritional information about various foods (e.g., calories, carbohydrates, 
fat, and vitamin content). Users have to scan the barcode on a food item or use the 
search box to find and manually add the specific food and item weight, and the 
application computes its total nutritional value. Chen et al. [10] reported that users’ 
quality of experience is much higher with smartphone application diaries than with 
pen and paper diaries. They also found that diabetic patients using application dia- 
ries reported a better food intake control than those using pen and paper diaries. 
Furthermore, a recent study by Bracken et al. [11] demonstrated that non-patient 
users wishing to lose weight (e.g., managing pre-obesity) and others wishing to gain 
weight (e.g., building muscle) utilize diary applications to attain their nutri- 
tional goals. 

Medical applications are oriented towards health workers and healthcare practi- 
tioners. These professionals can use these applications as a productivity tool in their 
work, which enables them to automate necessary tasks [12]. Recent work [13] has 
indicated the advantages of medical applications: they increase access to point-of- 
care tools, thus improving patient outcomes that stem from better clinical decision- 
making. Wattanapisit et al. [14] investigated whether a medical smartphone-based 
application can replace a general practitioner. They praised the use of an application 
for tasks such as recording medical history, making diagnoses, promoting health, 
performing some physical examinations, and assisting in urgent, long-term, and 
disease-specific care. However, the application was unable to support clinicians in 
performing medical procedures, appropriately utilizing other professionals, or coor- 
dinating a team-based approach. A recent literature review by Wattanapisit et al. 
[15] focused on medical counseling for physical activity and returned mixed find- 
ings regarding the usability and utility of medical applications. The review sug- 
gested that technical issues and the complexity of programs were barriers to 
usability, thereby implying the possibility of unfavorable patient outcomes such as 
inaccurate advice and diagnoses. 

Mobile network connectivity plays a significant role in always-online smart- 
phone applications. These applications may help to enhance an individual's 
decision-making and thus result in an improved QoL through connectivity to the 
Internet. However, such applications can also lead to the reverse effects. One exam- 
ple is smartphone addiction. According to the observations of Kwon et al. [16], “the 
overuse of smartphones can be easily seen in today's society" The examples pro- 
vided in the study include physical impacts (e.g., car accidents caused by smart- 
phone use) and mental impacts that create issues for smartphone-addicted children 
(e.g., a loss of concentration in class). The authors proposed the Smartphone 
Addiction Scale (SAS) to quantify this addiction. The SAS consists of 48 items 
relating to smartphone usage in distinct contexts, such as taking the smartphone to 
the toilet or feeling stressed when the smartphone is not connected to a network. 
Also derived from this scale is the Smartphone Addiction Scale for Adolescents 
(SAS-SV) [17], evaluated by the same authors. The SAS-SV was used by Haug 
et al. [18] in a study on young people in Switzerland, which found that social 
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networking applications were the applications most closely associated with smart- 
phone addiction. 

Smartphone addiction has also been attributed as a source of loneliness, poor 
bonding, and lack of integration, as shown by Bian et al. [19]. Samaha et al. [20] 
observed the relationships between smartphone addiction, stress, academic perfor- 
mance, and satisfaction with life. Through the use of multiple surveys, the SAS-SV, 
the Perceived Stress Scale, and the Satisfaction with Life Scale, they found addic- 
tion risk to be positively related to perceived stress. Finally, a large study by 
Carbonell et al. [21] demonstrated a substantial overlap between smartphone use, 
Internet addiction, and social media use in a student population. Smartphone addic- 
tion also has physical effects. For instance, Akodu et al. [22] described higher scap- 
ular dysfunction found in a population of students who are addicted to their devices. 

A considerable amount of literature has been published on the influence of 
smartphones and has found that smartphone applications may influence users’ 
QoL. Applications can contribute to users’ well-being both positively and nega- 
tively, depending on the applications used and the user profile. 


Smartphones as Sensors of Daily Life 


Research by Dey et al. [23] established that smartphones are within arm’s reach of 
their users an average of 88% of the time. Therefore, they are a beacon of one’s 
presence. Indeed, smartphones have been used during the COVID-19 pandemic as 
a proximity sensor for contact tracing [24]. In recent years, smartphones have 
become a critical tool for researchers in all fields, as one of the greatest challenges 
to conducting a study is collecting participants’ data. To solve this problem, a set of 
applications and software libraries have been developed to collect raw sensor data 
from smartphones as proxies for their users. These libraries collect similar data in 
different ways, although iOS devices are more restricted than Android devices. 

Smartphone data can be collected from the following onboard sensors: acceler- 
ometer, location, proximity, barometer, gravity, light, magnetometer, audio, and 
temperature. Communication data can also be recorded from Bluetooth, SMS, tele- 
phony, and social applications. Tools such as AWARE exist to simplify the data 
collection process [25]. However, AWARE is often unable to integrate with other 
software platforms, while other tools such as Sensus [26] have customization issues. 
Meanwhile, libraries such as SensingKit [27] cannot support data collection alone. 
Furthermore, other software platforms like the CARP Mobile Sensing framework 
[28] propose a multi-platform approach (Android and iOS) with a reusable UI 
(Flutter) and support sensing for numerous features, but they lack low-level, 
hardware-based, detailed information. 

Smartphone data collected with such tools have been successfully used in human 
studies [29]. For example, Ciman et al. [30] and De Ridder et al. [31] leveraged data 
collected from smartphone sensors to propose a stress assessment method. The first 
study used the data generated by finger swipes on the screen to detect stress, while 
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the second paper showed through a meta-analysis that a tailored smartphone appli- 
cation can directly extract the heart rate variability (HRV), which is a stress indica- 
tor, from images of the subject’s finger as it touches the smartphone’s camera under 
illumination from the smartphone’s flashlight. This process is called photoplethys- 
mography. Smartphones are also used as sleep duration sensors, which was explored 
by Ciman et al. [32], and can predict users’ intimacy, as claimed by Gustarini 
et al. [33]. 

In summary, smartphones are a proven source of daily-life data in multiple 
research domains, and their output has been validated experimentally. 


Mobile Network Connectivity Study: Methods 


QoL Lab was established in 2010, and since 2011, our research group has collected 
smartphone-based datasets for various human-based research studies and has used 
its own logging software for research into human activity recognition [34], mobility 
[35], and intimacy [33], among the other areas of study. The goal of this prior 
research was to quantify those aspects of human behavior with the use of smart- 
phone sensors (i.e., gathering data using accelerometers, gyroscopes, and network- 
ing information, for example) and participants’ self-reported inputs. We now focus 
on human subject studies “in the wild” and the practical aspects of smartphone data 
collection [36] through various research topics such as Quality of Service (QoS) 
[37], Quality of Experience (QoE) [38], and behaviors such as sleep [32] or stress 
assessment [39]. Smartphone data is collected in these different studies using the 
same framework (mQoL-Log), and it is tailored for each study. The mQoL-Lab 
application [40] enables background data collection through the mQoL-Log frame- 
work and implements surveys and remote notification to support human and 
smartphone-based research studies. Updates are necessary as the target system 
(Android OS) is always evolving. This section presents the tools used to acquire the 
data as well as their characteristics and discusses the selection of the derived fea- 
tures that are important for modeling individuals’ day-to-day mobile network con- 
nectivity. Furthermore, we detail the processes used for feature engineering and data 
filtering. 


Data Collection Periods and Overall Summary 
of the Collected Data 


We investigated participant connectivity with the use of mQoL-Log data records. 
We focused on the networking data collected through different studies conducted in 
Geneva over three time periods, which is presented in Table 23.1. Each participant 
was only present during their period. P1 was aggregated from a mQoL-Lab Living 
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Table 23.1 Data collection periods 


Number of 
participants (N) 
Pre/post-filtering 


References 
[41] 


Period ID Period Years 


2015-2017 


Study focus 
QoS, mQoL LLab 


P2 2018-2019 QoE, PeerMA [42, 38, 39] 63 55 
P3 2020 QoE [forthcoming] >] 3 
Total 121 110 


Missing 
Avg number of QI Q2 Q3 days 
Period | measurement days/ Standard days days | days avg x std. 
ID participant error Min | (25%) | (50%) | (75%) 
PI 168.6 15.4 6 79 187 270 322 | 59+10 
P2 30.4 22 4 24 29 32 98 |228z1 
P3 32.6 1.6 30 31 31 32 39 0+0 
Total 93.3 9.66 13.33 | 27 325 | 170 153 | 27243 


(mQoL LLab) observational study that focused on people's smartphone usage. P2 
studies were also “observational” and focused on quantifying the QoE of smart- 
phone applications. They also focused on stress assessment via peers (PeerMA 
[39]). The P3 study was the first “interventional study of smartphone application 
category recommendations made based on the QoE model”, where the intervention 
aimed to maximize user QoE in any context. 

The presented meta-study focuses on participants’ mobile connectivity through- 
out their days. The 121 participants collected a total of 69,761,823 samples. A sam- 
ple is a piece of timestamped network-related information that was collected 
automatically via the mQoL-Log either when mQoL-Log requested information 
(i.e. by pulling the network state every 60 s for P1) or when an event occurred, such 
as a handover between different network connection types (e.g. 4G to WiFi, net- 
work connection or disconnection for P2 and P3) being pushed to the logger. The 
different ways of collecting the networking data (push/pull) were dictated by the 
Google API changes over the years. A "day of the collection" is a calendar day 
(midnight-to-midnight) for which at least one sample exists. On average, each par- 
ticipant collected data for 85 days (= std. err 9), 21 days for the 25th percentile (Q1), 
31 days for the 50th percentile (Q2), and 128 days for the 75th percentile (Q3). We 
observed outliers in the aggregated dataset: one participant recorded 322 days of 
collection (max), while another only submitted one day of collection (min). Filtering 
was applied to the dataset following two exclusion criteria: (1) a participant col- 
lected less than ten samples, or (2) a participant collected less than three consecutive 
days of recording. The filtered dataset contained 110 participants; the filter removed 
11 participants and 18,550,170 randomly distributed samples. The remaining 
51,211,653 samples were retained for further analysis. Table 23.2 presents the 
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participation statistics for the filtered datasets collected in each period. A “day of 
measurement” is defined as any sample collected in a 24 h period during the collec- 
tion period; this is valid for P1, P2, and P3. Contrary to a “day of the collection", this 
new metric is not based on a calendar day but on the availability of samples in a 24h 
period (defined as a moving window or 24 h from a previous sample). 

For example, for the P1 participant who recorded 322 days of collection (max), 
we have defined 322 days of measurement, meaning that the time difference between 
any two samples was less than 24 hours and that at least one sample per calendar 
day (Monday, Tuesday, ...) was available. On average, each participant collected 
data for 93.3 days of measurements (+ std. err 9.66), 27 days for the 25th percentile 
(Q1), 32.5 days for the 50th percentile (Q2), and 170 days for the 75th percen- 
tile (Q3). 

Given that a sample is a piece of timestamped network-related information, if 
n >=] samples are generated at a specific minute (hh:mm), we classified this as one 
minute of data collected. Table 23.3 details the total number of minutes of data col- 
lected per period. We computed the mean rate of minutes acquired to understand 
how much data was collected per collection period overall. This rate differs from the 
days of data collection and the days of measurement, as it is minute based. We com- 
pared each sample acquired at a minute level to the possible number of data collec- 
tion minutes during the collection period, assuming zero data loss, i.e., with data for 
all the minutes available. The last column of the table shows the overall acquired 
minute rate over the three data collection periods. Compared to P2 and P3, as 
explained above, the data collected in P1 was acquired more frequently. 


Measurement Framework: mQoL-Log 


In 2011, within the context of the mQoL Living Lab, we developed the first version 
of a smartphone logger for the Android operating system, and we implemented a 
cloud-based infrastructure to collect smartphone data. The smartphone application 
was composed of two modules: the data logger (mQoL-Log) and the user interface 
(mQoL-Lab). The user interface contained the participant's communication medium 
to complete the study and provide the possibility to contact the study's principal 


Table 23.3 Number of minutes of data collected for the three periods 


Avg Standard | | QI Q2 | Q3 ] Mean acquired minute 
Period ID | [min] | error | Min | (25%) | (50%) | (75%) | Max rate + std.err [%] 
PI 86,148 9387.8 — 855 | 15,769 | 89,008 | 138,676 | 224,684 | 34 + 2 
P2 |1499 |151 [189 634 |1202 12021 |5559 |4+0.2 ! 
P3 |1370 |286 (546 | 1017 |1232 |1992 |2045 1320.5 


Total — (399705858 |536 1088 |2471 |75.339 |77429 |17+1.8 
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investigator. A cloud-based (our university-hosted) component was able to trigger 
surveys remotely and control the quality of the data collected on the smartphone, for 
integrity purposes. 

mQoL-Log collected the data from the smartphone as mentioned previously (see 
section “Smartphones as Sensors of Daily Life"). Table 23.4 presents data collected 
from the smartphone’s sensors through mQoL-Log. Table 23.5 presents in detail the 


Table 23.4 Data collected by mQoL-Log 


Trigger and 
Study frequency of 
Variable name | Definition period collection 
Screen The status of the smartphone screen and the user | P1, P2, Changes in screen 
activity interaction. P3 events (on, off, 
user presence, 
rotation) (push) 
Touches Number and duration of user touches on the Pl, P2, Screen event- 
screen during a usage session. P3 based: Each 
smartphone 
session (push) 
Active app Application name on the user screen Pl, P2, Changes in the 
name P3 application 
on-screen (push) 
Background | Application services running in the background | PI Every 60 s (pull) 
app (list) 
Connectivity | WiFi level, WiFi BSSID, WiFi SSID, WiFi PI Every 60 s (pull) 
and network | interface speed, cell ID, cell operator, cell P2, P3 Changes in 
strength, cell radio access technology (RAT), network 
cell network code, Internet connection status, connection state 
cell bandwidth up and down stream, number of and during user 
packets and bytes sent and received on wireless app usage (push) 
interfaces 
Round The RTT is the time needed for a ping to be sent | P1 Every 60 s (pull) 
Trip-Time by a smartphone to a server, plus the amount of | (always 
[ms] time taken for an acknowledgment to be unige.ch 
received. server) 
A ping is an active probing connection to a P2(app | When the app 
specific server via its address. A ping is server) usage session 
executed six times; the first is discarded to starts (pull) 
remove any noise from DNS resolution time. 
We derived statistics (mean, stdev, and variance) 
from five executions. 
Battery Battery status (e.g., charging, full, discharging), | P1, P2, Changes in 
battery level, battery temperature P3 battery state 
(push) 
Physical Physical activity of the user from Google Play  |Pl, P2, Changes in the 
activity Services activity (still, tilting: between two P3 user activity 
states, in-vehicle, on a bicycle, on foot, (push) 
running). 
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Table 23.5 mQoL-log network data 


Name Description 
Network type | Type of cellular or WiFi network (RAT). 


Signal strength | The signal strength is defined as the received power present in the WiFi and 
cellular radio in dBm (RSSI). dBms were transformed into the representation 
used in the Android OS, i.e. bars, as the participant would see this information 


on-screen. 
Operator The name of the cellular network operator. 
Unique Cellular network tower ID (cell ID) or WiFi basic service set identifier 


identifier (ID) | (BSSID). 
Network name | WiFi network name. 


Handover Flag indicating a change in network type, cell ID, or BSSID. 

Total Cumulative sum in bytes of downloaded data since the last smartphone reboot. 
downloaded 

data 


Total uploaded | Cumulative sum in bytes of uploaded data since the last smartphone reboot. 
data 


connectivity and network data collected. The logger included an energy policy to 
preserve the participant’s smartphone battery life by stopping all data collection at 
a threshold of 30% battery capacity. Collection resumed once the smartphone was 
charging or when the battery capacity was above the threshold. 


Final Dataset 


As we wished to compare the connectivity of participants for the given data collec- 
tion periods, we resampled the acquired P1, P2, and P3 datasets to one sample per 
minute, and completed the missing data points with the last known connectivity 
value. This method interpolates the missing points between two samples (upsam- 
pling), thus enabling a minute-based analysis of the smartphone’s connectivity. The 
following assumption was made to validate this dimension change (i.e., to discretize 
it to 1 min frequency): if no data is present between two samples, this means that no 
event occurred. With this, we propagate the last known value to the next minute 
until a different event-generated sample is found. However, we are fully aware that 
this process does not allow us to make generalizations about a representative sample 
of the population. 

Theoretically, P1 should have been sampled at one-minute frequency, since the 
pull method was leveraged for collecting the data every minute. However, we 
observed a skew in the pulling time, due to the Android OS giving lower priority to 
the collection process; the mean acquired pull rate was not 100% at | min period. 
Following the resampling process, the P1 was hence resampled to a one-minute 
frequency. As for P2 and P3, the resampling process generated a time series from 
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Table 23.6 Average measurement minutes collected post resampling, per participant in a period 


Mean 
acquired 
Period | Avg Standard | Min QI Q2 Q3 Max minute rate 
ID | [min] error | [min] (2590) | (50%) |(75%) |[min] + std.err [%] 
Pl 241,832 | 22,261 |7245 |113,091 | 267,415 | 388,762 | 462,466 | 100 +0 
P2 142,171 | 3144 [3931 | 33,402 | 39,637 | 45,089 | 139,726 | 100+0 
P3 45,122 |2473 41,448 |42,47 |42,930 43,902 [54,885 |100+0 
Total | 109,708 |9292 [17,541 |62,980 | 116,660 | 159,251 | 219,025 | 100 +0 


the discrete events collected by the push method. The total size of the resampled 
dataset is 234 million samples as presented in Table 23.6. 


Features Derived from Mobile Network Connectivity 


In this subsection, we describe the four features derived from the raw dataset: (1) 
network access technology, (2) signal strength, (3) data consumption, and (4) user’s 
physical mobility. 

Network access technology or radio access technology (RAT) is defined as the 
physical connection system for a radio-based communication network. Smartphones 
support several RATs, such as WiFi, Bluetooth, GSM, UMTS, LTE, or 5G NR (New 
Radio). The focus of this analysis lies on RATs that enable Internet connection, so 
the Bluetooth standard is out of scope. 

The signal strength is defined as the received power present in the WiFi and cel- 
lular radio signal. The signal strength feature directly impacts a user’s network con- 
text and provides an insight into the connectivity level at that moment to the current 
Internet provider (i.e., a cell tower or WiFi access point). 

Data consumption is defined as the amount of data (bytes) transferred from and 
to the smartphone through upload and download. The amount of data transferred 
during a specific time window provides information about the immediate network 
bandwidth. Some types of smartphone applications consume more data than others; 
for example, a video call application sends and receives more bytes than a text- 
based chat application. 

The fourth feature is the user’s physical mobility. Smartphones are used on the 
move, and their small size allows users to keep them in their pockets. In this way, 
they are a proxy for the user’s mobility. Mobile connectivity is dependent on the 
physical network infrastructure around the user. Therefore, we analyzed the mobil- 
ity aspect registered in the dataset. Mobility is defined as the number of cell towers 
and/or WiFi access points with unique identifiers that a participant passes through 
during a specific time window. 
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Network Access Technology 


Wireless network access technology on a smartphone consists of two Internet- 
enabled subtypes: WiFi and cellular. WiFi allows smartphones to connect to a wire- 
less local area network (WLAN). Often these local networks are also routed to 
provide Internet access. A smartphone’s WiFi interface connects to an access point 
(AP) to provide an Internet connection, which has a network name and a unique 
identifier. In contrast to a cellular connection, WiFi enables a smaller coverage 
range depending on the generation used (on the scale of meters rather than the kilo- 
meters of a cellular connection). For this reason, WiFi is primarily used to connect 
to the Internet from home, work, or university. Various generations of cellular net- 
works have been developed (e.g., 3G, 4G, 5G) with the evolution of access technol- 
ogy (see Table 23.7). 

A cell tower offering Internet connectivity also has a unique identifier, but the 
main differences between cell-based technologies generation are the speed of the 
connection and their coverage range from the antenna. A smartphone’s baseband 
processor is the chip on its motherboard, which manages all radio functions. This 
processor is separated from the main smartphone processor for three reasons: (1) 
radio performance: the main processor is too slow to handle the type of work done 
by the baseband processor, such as encoding and modulation; (2) legal: authorities 
require the software that manages radio transmission to be certified; and (3) 


Table 23.7 Generation of cellular network access technologies 


Max Estimated download time 
download for a 3 min 1080p 
Generation | Acronym | Full name speed YouTube video (75 MB) 
2G GPRS General Packet Radio 0.0125 Mbit/s | 800 min 
Service 
EDGE Enhanced Data Rates for 0.0375 Mbit/s | 27 min 
GSM Evolution 
3G UMTS Universal Mobile 0.0375 Mbit/s | 27 min 
Telecommunications 
System 
HSPA High Speed Packet Access | 0.9 Mbit/s 11 min 
HSDPA  |High Speed Downlink 14 Mbit/s 1.1 min 
Packet Access 
HSUPA | High Speed Uplink Packet | 14 Mbit/s 1.1 min 
Access 
HSPA+ | Evolved High Speed Packet | 42 Mbit/s 13.8 s 
Access 
4G LTE Long-Term Evolution 150 Mbit/s 0.001 s 
(Cat4) 
5G NR New Radio 400 Mbit/s 15s 
(sub-6Ghz) 
1.8 Gbits/s 
(mm Wave) 
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reliability: the OS or new application versions should not interfere with the base- 
band processor functions. The baseband processor is the component that manages 
the handover between network access technologies. When a tower is located too far 
from a smartphone, the signal may drop and end the user’s connectivity. The base- 
band processor then automatically connects to a closer antenna to provide network 
access. If an antenna is not available in the same RAT, the baseband processor, 
selects a lower technology RAT, as older RAT often provide a larger range of cover- 
age. For instance, if a 4G signal is unavailable because the user is on the move, and 
no other 4G link can be established, the smartphone will attempt to connect to a 3G 
antenna. 

The type of network access technology is important because it is directly linked 
to the quality of connectivity. As Table 23.7 shows, an EDGE-based connection 
theoretically has a maximum download speed of 0.0375 Mbit/s, which is not enough 
to watch a YouTube video [43]. WiFi technologies have also undergone several 
stages of evolution with different maximum download speeds, i.e., WiFi type (e.g., 
a, b, g, n, ac). However, this information was not available during dataset collection, 
so information regarding WiFi speed is not included in this analysis. Connection to 
a WiFi network is not automatic, as the user must enter credentials to connect to the 
network. These credentials ensure the encryption of the communication between the 
smartphone and the wireless AP. The credentials exchange is transparent on a cel- 
lular connection, in which case the baseband processor communicates with the 
Security Information Management (SIM) card and the operator network to authen- 
ticate the smartphone on the network. 


Signal Strength 


We examined the overall network connectivity signal strength over the collection 
periods. Signal strength is always presented on the smartphone screen and is 
located in the upper right-hand corner on Android and iOS. Icons represent the 
signal strength sensed by the onboard antennas for both WiFi and cellular networks 
in a human-readable format. The mQoL-Log application was able to collect that 
information in decibel-milliwatts (dBm). To utilize this information, we deter- 
mined how the Android OS presented this data to the end-user, and mapped the 
dBm to the number of bars (0 to 4) shown on-screen. The signal strength represents 
the power present in the received radio signal. For smartphones, this directly 
impacts the QoE of smartphone services such as video streaming and online games. 
The minimum signal strength needed to achieve a “good” experience when watch- 
ing an online video on the move depends on the network access technology and the 
video format (e.g. HD or 4 K). The signal strength plays a significant role during 
handovers. The baseband processors collect the signal strength continuously and 
choose whether to switch between RATs (i.e. conduct a vertical handover for the 
same RAT or a horizontal handover if RATs change) or between cell antennas. 
Connectivity-wise, the smartphone user sees the signal strength as an overall health 
indicator of the network connection. Thus, a user may decide not to start a video 
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call if the smartphone reports low signal strength, instead preferring to communi- 
cate via audio call only. 


Data Consumption 


Data consumption is a significant feature in the context of connectivity. The RAT 
limits the amount of data that can be transmitted, measured in seconds. Accordingly, 
the amount of data consumed is bound to the current network access technology. 
The data consumption depends on the type of services utilized by the smartphone 
user. Video applications consume a large amount of data, e.g. by downloading 
video, while a video calling application simultaneously generates and consumes a 
large amount of data by uploading and downloading a video. The overall data con- 
sumption also provides insight into the network traffic state. If the network encoun- 
ters a large amount of traffic, this impacts the bandwidth available for use in a live 
video or other application by the user, and the user connectivity is affected. The 
amount of data downloaded and uploaded also indicates the user profile type, as 
some users consume less data than others. This may be due to the nature of their 
subscription to their operator (financial), the services used on their smartphones 
(behavior), and the quality of the link connecting them to the Internet (structural) 
over time [44]. 


User’s Physical Mobility 


User mobility is essential, as discussed previously. Indeed, connectivity and mobil- 
ity are crucial to understanding participants’ smartphone usage and connectivity 
changes. We explored participants’ mobility per hour and the number of times each 
participant connected to the same tower or the same AP for multiple periods (days 
to weeks). A large number of unique identifiers (ID) is an indication of high mobil- 
ity for a participant. One cell tower covers a few kilometers of land in a densely 
populated area (e.g. a 4G tower has a 16 km range), while a WiFi AP covers only a 
few meters (e.g. a WiFi ac reaches 12—35 m inside and up to 300 m outside). 


Mobile Network Connectivity: Results 


We analyzed results for the four features that quantify the connectivity level of an 
individual relying on the connection and usage of their smartphone network: the 
network access technology (section “Network Access Technology"), its signal 
strength (section “Signal Strength"), overall data consumption (section “Data 
Consumption"), and mobility (section “Users’ Physical Mobility"). For each 
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feature, we present the overall statistics (post-filtering) of the 110 participants orga- 
nized by their respective collection period. 


Network Access Technology 


Table 23.8 presents the overall average of RAT distribution per measurement period. 
Figures 23.1, 23.2, and 23.3 illustrate the distribution of network access technology 
for P1, P2, and P3 participants, respectively. The figures clearly show the adoption 
of LTE (4G). In the P1 distribution, we observe a high presence of HSPA, while the 
P2 distribution suggests that some participants (particularly P2898 and P2S64) 
were not connected (NOCO) for the majority of the study. Overall, we see lower 
access to the Internet in P2 than in P1 and P3. The most recent data demonstrate the 
rise of LTE and WiFi over the RAT. Furthermore, during P3 the participants had the 
most stable connection to the Internet (low NOCO), as presented in Table 23.8. 

Figure 23.4 presents the overall average distribution over the three periods. We 
observe that LTE is more present than WiFi in P3. 

The data imply that overall, on average for all periods, any connection to the 
Internet is present 93 + 0.8% of the time (averaging 104,540 + 64.36 min across all 
periods). This information is computed from the RAT distribution. Table 23.9 pres- 
ents the distribution of the connectivity and the average minutes of connection for 
each period and reveals that P2 connectivity is lower than that of P1 and P3. 


Table 23.8 Overall average RAT distribution (76) per data collection period 


RAT/period 
[20] P1 P2 P3 
WiFi 52.1 33.7 51.0:+3.2 47.7+7.1 
LTE 29.8 + 4.1 28.7 + 2.9 49.8 x 0.5 
HSPA+ 3.1 € 0.8 33212 0.8 x 0.5 
HSUPA 0.1+0.1 0.0+0 0.0+0 
HSDPA 22x12 0.0+0 0.0+0 
HSPA 3.9+41.1 0.4+0.1 0.1+0.1 
UMTS Zll 0.7 + 0.3 0.3 + 0.2 
EDGE 2.0 + 0.8 0.6 € 0.2 0.2 x 0.1 
GPRS 0.0 € 0 0.0 + 0 0.0 + 0 
GSM 0.0 x 0 0.0x0 0.00 
UNKNOWN 2.6 + 0.9 0.5 0:3 0.342 
NOCO 1.0+0.2 14.7 + 2.4 0.7 x3 
Download speed on cell network in Mbit/s 6.9 x 3.3 6.4 x3 9.4 € 4.6 
Avg x std.err 
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Fig. 23.4 Overall average RAT distribution over P1, P2, and P3 
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Table 23.9 Percentage of connectivity to internet distribution per data collection period 


Connectivity (%) per period PI P2 P3 
Mean 0.96 0.85 0.99 
Std 0.06 0.18 0.00 
Min 0.69 0.12 0.99 
25% 0.97 0.77 0.99 
50% 0.99 0.92 0.99 
7596 0.99 0.97 0.99 
Mean in 233,162.51 35,775.89 44,682.18 
Minutes/total time/in period + £ + 
+ std.err 21,371 2672 2448 
Signal Strength 


The temporality of signal strength for each group is presented in Fig. 23.5. Signal 
strength increased with time for each group. P1 and P2 feature homogenous signal 
strength, in contrast to P3, which exhibits a higher signal strength at weekends and 
during mornings. 

Figure 23.6 presents the overall signal strength distribution per period. The resa- 
mpling process explains the high prevalence of the 0 bar. 

Figure 23.7 presents the correlation between the signal quality and the connec- 
tion type over all three periods. We note a high degree of correlation between WiFi 
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Fig. 23.6 Overall signal strength distribution per data collection period 
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Fig. 23.7 Pearson Correlation Between Signal Strength and Network Access Technology Type 


and signal strengths of | and 2 bars, while LTE network technology and signal 
strengths of 3 and 4 bars display a moderate correlation. 


Data Consumption 


During the analysis, we observed high data consumption by particular participants, 
as depicted in Fig. 23.8 with the cumulative distribution function (CDF) for monthly 
data usage and in Fig. 23.9 with the daily data usage for each participant (each data 
point on one of the lines corresponds to a participant). In both figures, each data 
point represents the average monthly data consumed by one study participant in 
terms of (1) rx (received, downlink) and (2) tx (transmitted, uplink), overall and for 
cell-based networking. The majority of the participants display similar data- 
consuming behavior, regarding both data receiving and transmitting. In both tempo- 
ral modalities, the amount of data transmitted from the smartphone to the cellular 
network is lower than the amount of data received. The monthly and daily data 
usage follows the same pattern (Fig. 23.8), while we observed faster consumption 
in the daily data usage (Fig. 23.9), in both figures each sign represents a 
participant. 

Figure 23.10 presents the min — max-normalized weekly mean data received 
from all participants over the three periods. A larger amount of downloaded data can 
be observed during the weekends compared to the rest of the week. Participants 
consumed more data during mornings and evenings, and downloaded more data on 
weekends. We observed clusters of spikes during afternoons and evenings. The P3 
participants received fewer data during the weekend. As expected, a low volume of 
data was received by smartphones during the night. 
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Fig. 23.8 Monthly data 
usage CDF (all Periods) 


Fig. 23.9 Daily data usage 
CDF (all periods) 
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Fig. 23.10 Normalized weekly mean amount of data received per data collection per period 


We found that participants in P2 consumed more data than the other cohorts, as 


shown in Fig. 23.11. P3 data consumption is less sparse, likely due to the number of 
participants in this cohort. In all three periods, we observed some outliers that con- 
sumed more data than other participants. 
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Fig. 23.11 Total downloaded and uploaded bytes per period normalized by number of data collec- 
tion days 


Users? Physical Mobility 


We focused the analysis on the number of individual cells and AP IDs. The full 
dataset contains 59,602 unique cell IDs and AP IDs combined. It is important to 
note that the same Wi-Fi network can be accessed via different APs, in which case 
the ID is different, but the network is the same. This enables roaming between the 
different APs in the same domains. This type of configuration is often found in large 
networks, for example in companies, universities, and large houses. In such cases, a 
Wi-Fi repeater is installed to obtain better signal quality over the entire area. The 
repeater has the same Wi-Fi network name as the main AP, but it has a different 
ID. Like smartphones, these devices reconnect to another AP when they lose a con- 
nection, such as when the user is on the move. 

The vertical handover process is seamless, and the device automatically recon- 
nects to a Wi-Fi network that shares the same name as the previous network. In this 
case, the device already knows the security configuration to obtain a secure connec- 
tion, namely a previously established authentication. Figure 23.12 shows the mean 
cumulative cell tower and Wi-Fi ID changes per hour and per day of the week, 
normalized from 0 to 1. In Fig. 23.12, we observe a lower number of unique IDs on 
Sundays for all periods. Other patterns are present; for instance, on Friday evenings 
participants were highly mobile, and the reverse is found during the night. P3 dem- 
onstrates lower mobility on Saturday evenings than P1 and P2. One possible expla- 
nation is the data collection time; P3 was recorded after the end of the first 
partial-lockdown in Switzerland during the COVID-19 pandemic. At this time, par- 
ticipants would have been less inclined to participate in external social gatherings 
on two consecutive nights. 
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Fig. 23.12 Mean cumulative cell Tower and Wi-Fi AP ID Changes per Data Collection per Period 


Discussion 


The results confirm our hypothesis that network connectivity and consequently the 
mobile Internet is widely available in today’s developed world. The results indicate 
that the participants’ smartphones were connected to the Internet for 93% of their 
day (+0.8%) on average. Their devices were always either connected or searching 
for new network access via Wi-Fi APs and cellular towers. The quality of the con- 
nection was high overall, and we found a strong correlation between LTE and high 
signal strength. Furthermore, as data quantity is directly connected to the services 
used on the smartphone and the available network bandwidth, we observed multiple 
data consumption patterns that could be used to profile the users. Taken together, 
these findings provide important insights into the four features that impact users’ 
connectivity and may influence an individual’s decision-making and consequently 
their QoL. In this section, we discuss the results and their limitations before recom- 
mending other data sources for modeling environmental QoL via connected 
services. 


Discussion of Overall Results 


Over the data collection periods (2015-2020), the adoption of 4G (LTE) network 
access technology was close to complete in the Geneva area. The low presence of 
network access technologies other than LTE and Wi-Fi in P3 can be attributed to the 
continuous efforts of the mobile operators in updating network infrastructure (1.e., 
new antenna deployment), an update in performance of the smartphones' baseband 
processor (i.e., which leads to a faster handover), and the low mobility of the partici- 
pants. A participant would have a higher number of connections if they were more 
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mobile. Furthermore, contrary to the data in P1 and P2, data from P3 was acquired 
during a shorter period of time from a smaller sample size. 

We found a strong prevalence of Wi-Fi usage during all three periods. As Wi-Fi 
is commonly used at home and at work, we made the assumption that Wi-Fi usage 
occurs mostly indoors, where participants are located. Furthermore, while Wi-Fi 
connection costs are not linked to the amount of data consumed, this 1s not the case 
for cell-based connections. Wi-Fi is generally provided by a broadband Internet 
connection. As noted in section *Network Access Technology", some participants 
use Wi-Fi less than others, possibly for cost and quality reasons. The cost of broad- 
band is high in Switzerland, and it is cheaper to obtain an unlimited 4G connection 
than to have both a (Wi-Fi) broadband connection at home and a 4G subscription. 
The broadband connection quality also plays a role; if an area has a low population 
density, broadband operators will not invest in high-throughput infrastructures. As a 
result, using a smartphone's 4G connection to provide home Internet may become 
convenient and financially attractive. 

Our results introduce an additional reflection with respect to the cellular and Wi- 
Fi connectivity, and especially the handover between the two. Autonomous hando- 
ver between cell-based networks and Wi-Fi has not always been possible in 
smartphones. However, smartphone OSs have evolved and can now automatically 
switch between a Wi-Fi and a cell-based connection. In fact, the switching between 
the two types of connection is common in everyday smartphone usage. For exam- 
ple, after entering a home, a smartphone will automatically connect to the home's 
Wi-Fi router. A smartphone will switch to cell-based connectivity if the Wi-Fi con- 
nection is of low quality. This so-called smart assist feature is totally transparent to 
the user and does not require interaction with the smartphone. However, this process 
only operates in one direction (i.e., Wi-Fi to 4G); the smartphone does not subse- 
quently test the Wi-Fi network to attempt to revert to the Internet’s connection 
source. Given our results, we would recommend that the smart assist feature operate 
both ways. 

Additionally, connection and disconnection events to a cell tower are important 
data for a network operator. Notably, operators ultimately use the data collected 
from their core network, particularly the number of smartphones connected to an 
antenna, to generate connectivity maps and understand how to improve their ser- 
vices. Indeed, the services can also be improved by the network operator by enabling 
better connection during times of increased demand for connectivity in a given area. 
Conversely, the network operator could also enable a low-power mode of their sys- 
tem during low data consumption hours, decreasing their standby energy consump- 
tion. As shown in section “Data Consumption”, we found the same pattern as 
Walelgne et al. [45]: low data consumption during the night and higher consumption 
during the evening and early morning. These patterns reflect how people use their 
devices and connectivity. The observed higher throughput could originate from 
video consumption (leisure) or video conferencing with loved ones. This informa- 
tion can be used by a network operator to rent more bandwidth from its network 
provider, thus enabling a high-quality video conferencing experience at a spe- 
cific time. 
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This study has several limitations. The populations of participants in the three peri- 
ods are not identical, so we are unable to comment on the evolution of the individual 
populations. Additionally, the two main OSs for smartphones are Android (Google) 
and iOS (Apple), but data was only collected from Android users in this study. As a 
result, information and insights about the population of 1OS users is missing from 
this study. Additionally, the number of participants and the duration of P3 is lower 
than that of P1 and P2, so the generalization of the results between the cohorts is 
limited. We encountered another limitation during data logging due to the short- 
comings of the OS de-prioritizing our logger. With the P1 dataset, we found that it 
was impossible to collect at least 5096 of minute-based samples, even by sampling 
with the minute-based pulling method. Future studies shall be designed such that 
they address these limitations. 


Quantified Self Movement 


The Quantified Self (QS) movement brings together individuals from different 
backgrounds who wish to learn about themselves. The QS practitioners use tools, 
principles, and methods that are mostly enabled by smartphone applications and 
services and allow them to measure, analyze, and share their data [46]. The QS tools 
can include medical test results or well-being-oriented connected objects (e.g., fit- 
ness trackers, smartwatches), mobile applications, and web applications. Those 
sources of information can also contribute to collecting a high-dimensional connec- 
tivity dataset and data to quantify individuals’ behaviors, health, and QoL [47]. 
Currently, the QS practitioners are mostly interested in their habits and health. They 
collect large amounts of data that they usually openly share on online platforms 
(e.g., quantifiedself.com, openhumans.org) for others to experiment with. In doing 
so, they expect to learn about themselves through their own analyses and through 
others'. 

In the QS movement, smartphones are the main collection devices (c.f., Chap. 1). 
For instance, diary and reminder applications are often deployed to collect one's 
day-to-day emotions, mental states, social interactions, and other aspects of human 
life currently unquantifiable via autonomous, connected devices. Those devices and 
applications depend on mobile network connectivity to function. However, the col- 
lection of network connectivity by the followers of the QS movement is often 
neglected. At the same time, smartphone data loggers that collect smartphone user 
habits, such as mQoL-Log, are uncommon in QS. In the future, it will be important 
to explore the potential use of additional data sources in the QS movement such as 
smartphone connectivity levels and their influence on the daily life of the individu- 
als. Knowledge and anecdotal data (i.e., a study with one participant) obtained by 
QS's followers could prompt further investigation by researchers into the links 
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between mobile network connectivity, physical health, social iterations, individuals’ 
overall decision-making, and QoL, for example. 


QoL Technologies 


The evolution of QoL technologies (QoLT), defined as technologies that enable 
assessment and assurance of life quality for individuals [48], is deeply linked to the 
development of individuals’ connectivity. The possibility to improve one’s life with 
QoLT would likely involve a component of communication to the Internet (e.g., a 
cloud) or edge network devices. The large amounts of data produced by personal 
wearable health sensors and smartphones, for example, would be processed for 
immediate use (in emergency situations) or for later use. The degree of QoS offered 
by QoLT would depend on the supported mobile network connectivity level. 
Therefore, the four features described in this chapter are important, as they are fun- 
damental aspects that define the individual’s connectivity. Without connectivity, 
there may be no QoLT. To elaborate on this point, we discuss QoL aspects defined 
according to the WHO and connectivity-dependent services. 

The domain of physical health includes many important facets, including daily 
living activities. Some of these activities rely on indoor connectivity being provided 
in the home or at work, school, or other frequent locations. The activities may 
require a low-latency, high-throughput network connection to operate. For instance, 
smartphone applications can provide medication schedule reminders and notifica- 
tions to a patient and their family. Energy, fatigue, and mobility are factors that can 
be quantified by smartphone and wearable data, and adequate real-time personal- 
ized care services can be provided to the individual, depending on their needs. The 
applications can also help a population with substance dependence issues; for 
example, some applications can put at-risk individuals in real-time communication 
with medical professionals. In the case of assisted living, connectivity can enable 
support services like remote healthcare and, in the future, robot care. Overall, many 
day-to-day physical health services provided to an individual in a given context can 
be supported by connectivity. 

The psychological health domain of QoL may be influenced positively or nega- 
tively by smartphone applications. Connectivity to services through smartphone 
applications can contribute to improving this domain. Services that influence this 
field include entertainment (e.g., watching a video), which can influence feelings, 
and information services (e.g., reading news on social media), which can influence 
thinking processes. 

In the social relationships domain, services enabled through an Internet connec- 
tion can range from simple text-based messaging to smartphone-based video con- 
ferencing. More generally, opportunities for social relationships provided by 
connected services are extensive and are evolving. These services may range from 
interactive entertainment services (e.g., joint use of online games, which influences 
feelings) and social networks (i.e., communication and exchange of information, 
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thus influencing the quality of the relationship). The sex industry understood this 
potential market and created multiple devices for remote sexual interaction through 
the Internet, providing intimacy for long-distance couples [49]. In the social rela- 
tionships domain, the specific challenge is to ensure sufficient mobile network con- 
nectivity for both receivers to enable content exchange with sufficient user 
experience during the interaction. 

The features of the environmental domain of QoL may be difficult to quantify, as 
it contains the most facets of any QoL domain and is influenced by contextual vari- 
ables that may not yet be understood. For example, opportunities for leisure or edu- 
cation may involve the possession of interactive entertainment (and a joint use of 
devices such as smart TVs, for example, thus influencing feelings), the use of social 
networks, or the use of online education services (e.g., services for peer communi- 
cation and the exchange of information). Because of the high interactivity of these 
examples of online leisure and education opportunities rely on connectivity to suc- 
ceed. Overall, there are many services in the users’ environments that may enable a 
better QoL and rely on mobile connectivity to be provided. However, the challenge 
is that a unified, well-understood model of these services and their connectivity 
does not exist yet. 

In conclusion, QoLT may impact all the QoL domains in beneficial and detri- 
mental manners, all depending on the implementation of the services it supports. 


Conclusion 


This chapter quantifies the mobile network connectivity of individuals in the Geneva 
area during three data collection periods between 2015 and 2020. Our results dem- 
onstrate that connectivity is ubiquitous in the day-to-day life of the participants of 
this study, as they could access their online services anytime and from any location. 
We also observed a time-based evolution of the participants’ Internet connection 
throughout the day. Overall, our results suggest that connectivity in the same geo- 
graphic location improves over time. The explored features (signal access technol- 
ogy, signal strength, data consumption, and users’ physical mobility) offer some 
insights into the participants’ connectivity. 

We observed a high correlation between signal strength and several network 
access technologies. According to our data, on average, a better signal strength is 
available on LTE than on Wi-Fi. Furthermore, knowing the individual data con- 
sumption patterns (amount of data received and transmitted) permits the profiling of 
study participants. Users who consume more data during a short period (spike) may 
use services that other users may not access because of their low connectivity. 
Additionally, we considered the amount of data received and transmitted by the 
smartphones at different times of the day. Although we found peaks during the eve- 
nings for P1 and P2, P3 did not exhibit this pattern. It is possible that a large amount 
of data consumption was taking place on other devices for a better experience dur- 
ing the evening (e.g., watching YouTube videos on a television screen instead of a 
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smartphone screen). In addition, we also observed less mobility on Sundays across 
all periods. We compared the overall mobility in all periods and noticed a lower 
mobility in P3, which was possibly due to the COVID-19 situation in Switzerland 
at the time of the study. 

We discuss the results in the context of emerging QoLT, which, embedded in 
personal devices including wearables and smartphones, enable the collection of 
health information, which may support an individual’s progress towards better 
health behaviors and, consequently, a better QoL. Overall, an increase in the use of 
QoLT may contribute to a better life. The range of services provided by QoLT rely 
on network connectivity, so future research work is needed to ensure that this con- 
nectivity matches the requirements of the technologies anywhere the user may be at 
any time. 
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Introduction 


Transportation is an essential functionality. Individuals perform routine activities to 
meet their specific daily demands, and transportation provides the opportunities to 
perform these, as well as to attend to a variety of commercial and social activities, 
amongst others. For instance, working individuals commute to/from work, children 
get to school or take part in extracurricular activities, families participate in leisure 
activities, and the elderly population engage in social or voluntary gatherings. 
Taking part in these activities not only enhances social interaction but also contrib- 
utes to the physical and emotional wellbeing of individuals [1, 2]. Studies show that 
the perceived quality of public transportation [3, 4], physical mobility [5], participa- 
tion in recreational travel [6], residential relocation [7] and active travel (walkabil- 
ity: [8] and cycling: [9]) affect quality of life. 

Conversely, negative impacts of transportation causes individual quality of life 
(QoL) to deteriorate. For instance, an increase in vehicle kilometers traveled (VKT) 
by private motorized vehicles negatively affects the environment through vehicle 
emissions and traffic congestion [10]. These traffic problems cause increased costs, 
travel delays, health impacts, air pollution and a reduction in individual subjective 
wellbeing (SWB) [11]. A similar study reported that long commuting to work 
induces stress and impacts psychological wellbeing [12]. 

In transportation-based QoL, most studies have mainly focused on SWB mea- 
sures to improve the transport mobility of specific groups. For instance, many stud- 
ies address SWB for transport disadvantaged groups with a special emphasis on the 
aging of society [13-18]. To date, the holistic approach to understanding the asso- 
ciation between various aspects of transportation and the SWB measures has not 
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received enough attention. Against this background, the central question addressed 
in this paper is how do various transportation aspects affect QoL and how can these 
aspects be quantified in the individual’s daily life? What are the relevant transport 
policies and practices that can enhance wellbeing at the individual and soci- 
etal level? 

The remainder of this chapter is organized as follows. Section "Transportation 
Related QoL” presents the findings of the scoping and expert consensus and identi- 
fies the extant literature that explores transport-related indicators in the context of 
QoL. Following this, the conceptual framework (TRAWEL) is then presented in 
section “TRAWEL: A Conceptual Framework for Transportation Based QoL”, 
explaining the link between five broader dimensions of transport (transport infra- 
structure and services, built environment, transport externalities, travel-based time 
use and travel satisfaction) and measures of SWB (community, social, economic, 
physical and psychological). The indicators, measures and methods used for the 
continuous, longitudinal and quantitative assessment of these five aspects in a well- 
being context are discussed in section “Quantifying Transportation: Method and 
Measures". Finally, the chapter discusses its contribution in section "Discussion" 
and concludes by surveying policy implications towards improving wellbeing and 
future research directions in section “Conclusive Remarks". 


Transportation Related QoL 


The World Health Organization [19] defined individual QoL as “individuals’ per- 
ception of their position in life in the context of the culture and value systems in 
which they live and in relation to their goals, expectations, standards, and con- 
cerns”. Individual QoL in the context of transport mobility is defined by the WHO 
as "the person's view of his/her ability to get from one place to another, to move 
around the home, move around the workplace, or to and from transportation ser- 
vices". Additionally, Myers [20] state that *a community's QoL is constructed of the 
shared characteristics of the residents' experience in places (for example air and 
water quality, traffic or recreational opportunities) and the subjective evaluations 
residents make of those conditions". 

The central focus of this chapter is how transportation alters the QoL of an indi- 
vidual. To this end, a number of studies were critically examined to understand the 
association between transportation aspects and wellbeing. Methods leveraged in 
this paper include the scoping review method and expert opinion consensus. Articles 
and grey literature published from January 2005 to June 2020 were identified, 55 of 
which met the inclusion criteria and are presented in this paper. The inclusion crite- 
ria stated that a study had to focus on one of the seven performance indicators of 
transportation in the wellbeing context: mobility, affordability, accessibility, con- 
nectivity, externality, travel needs and attitudes. In the following paragraphs, we 
define and discuss these separately. 
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Transport mobility is defined as the ability to move from one place to another 
using different types of movement such as walking, cycling, transit and driving [21, 
22]. Studies on transport mobility broadly focus on measuring the impact of the 
transport sector on the QoL of the elderly population, especially in developed econ- 
omies [2, 23]. Several studies highlight that out-of-home mobility positively influ- 
ences the QoL of aging individuals. Older people (>65 years) with a driving license 
are likely to enjoy better mobility benefits in terms of out-of-home activities com- 
pared to non-drivers [23]. Social exclusion occurs when the transport mobility 
needs of elderly people are not adequately addressed. For instance, Jalenques et al. 
[24] found that older people who do not drive tend to be adversely affected with low 
QoL due to their dependence on others. Besides older people, Jones et al. [25] 
explored the impacts of micro mobility i.e., e-bikes, on individual health and well- 
being. Based on qualitative assessments, the purchase and usage of e-bikes posi- 
tively correlate with personal wellbeing. Moreover, pedestrian-oriented development 
plans such as walkability and bike ability services also positively impact individual 
QoL [26, 27]. 

Transport affordability as defined by Litman [28] is the ability by all house- 
holds to make journeys and access services while devoting less than 20% of house- 
hold budgets to transport. In this manner, studies have analyzed the economic aspect 
of transportation using QoL indicators to understand individual wellbeing. For 
instance, De Groot and Steg [29] examined the impact of transportation pricing 
policy on car use using 22 QoL indicators (e.g., comfort, money, income and envi- 
ronmental quality). The study found that cost of car use negatively affects certain 
wellbeing indicators such as environmental quality, money, change and work, while 
it positively impacts indicators such as comfort and safety. Using the Delphi method, 
Zelinkova [30] assessed the effect of road pricing on QoL measures such as safety 
and stress and found that successful transportation implementation depends on 
political will and public acceptance. Schwarzlose et al. [31] found that users are 
willing to pay for a flexible transport route service to improve QoL for the rural 
elderly. 

Transport accessibility is defined as “the extent to which land-use and transport 
systems enable individuals to reach activities or destinations by means of a (combi- 
nation of) transport mode(s)” [32, p. 128]. The accessibility of various facilities 
around the neighbourhoods is measured by the urban density, diversity of neigh- 
bourhoods, land use mix, green space, open spaces, walkability and connectivity. In 
particular, Ritsema et al. [33] suggested that living in high-density environments 
enables greater transport accessibility than in low-density suburbs. Other studies 
have conceptualized QoL dimensions to enable transport accessibility in transport 
planning. Lee and Sener [34] developed the Transportation QoL (TQoL) framework 
encompassing four dimensions of physical, mental, social and economic wellbeing 
and three components of the transportation system, namely accessibility, the built 
environment and vehicle traffic. In another study, Nakamura et al. [35] categorized 
transportation access, amenity and safety as QoL dimensions for understanding 
residential choices focusing on Transit-Oriented Development (TOD) for different 
socioeconomic groups. 
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Transport connectivity focuses on the links of the entire system that represent 
the interaction between multimodal transport modes and the ease of access to them 
[36]. Haslauer et al. [37] explored how the proximity and connectivity of public 
transport services within a community enhance QoL. Other studies have suggested 
that public infrastructure, connectivity, public space and green space positively 
influence the wellbeing of residents and thus enhance the neighborhood QoL [38, 
39]. In addition, a series of studies on the built environment has confirmed that land- 
use heterogeneity characterized by mixed land use, walkability and park density 
enhances social connectivity and thus improves community wellbeing, while popu- 
lation density and proximity to mass-transit stations negatively impact health- 
related QoL [40-44]. 

Traffic safety and air quality are major concerns of road transport planning that 
directly and indirectly affect health [45]. A series of studies on traffic safety demon- 
strates that overall QoL decreases in cases of injury and illness [46—49]. In addition, 
Putra and Juwita [50] analyzed how a public transport service with low safety stan- 
dards increases stress for passengers. Regarding personal safety, studies have found 
that perceived safety on urban streets affects QoL. In this context, Deegan and Baker 
[51] assessed residents’ perceptions on road infrastructure and find that dark and 
narrow streets, illegally parked cars and low social cohesion affect personal safety. 
Also, Panek et al. [52] reported that residents fear crime in underpasses, on train 
stations and on dark and narrow roads. Besides safety, traffic-related pollution 
directly affects SWB [53]. It has been found that exposure to traffic-related air pol- 
lution impacts physical wellbeing, while subjection to noise annoyance affects psy- 
chological wellbeing [54]. 

Travel needs are derived from different types of activities and vary not only 
from person to person but also with different life stages [55]. Many travel time-use 
studies have found that travel serving social interaction or recreation enhances both 
physical and mental wellbeing [18, 56-61]. Job and job-related travel causes stress 
and negatively impacts wellbeing due to the congestion, crowding and unpredict- 
ability of peak time travel [62-65]. 

Travel related attitudes refer to the psychological evaluation of transport sys- 
tems and daily travel elements (e.g., travel modes, trips and travel time), conveying 
some degree of favor or disfavor [66]. To understand the impact of changing travel 
behaviour on wellbeing, studies evaluate the association between travel attitudes 
towards travel choices (mode use, commute time, route destinations) and emotional 
wellbeing. For instance, studies have quantitatively investigated how attitudes 
towards commute trips by different modes or travel time influence hedonic and 
eudemonic wellbeing [67-71]. These studies suggest that shorter travel times and 
active travel modes positively affect hedonic wellbeing in the short-term and induce 
an eudemonic state in the long run. 

Summarizing the research results so far, the following research gaps are observed 
in the quantification of QoL dimensions in transportation. First, in addressing the 
impact of transportation infrastructure and services, it may be noted that relatively 
little research has examined the impact of mobility and affordability on QoL for 
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economically disadvantaged social groups. For instance, differences in household 
income levels affect individual and household wellbeing, also affecting mobility 
and the affordability of transport services, as pointed out by Hernandez [72]. 
Existing studies fail to address the impact of reducing transport costs in terms of 
vehicle purchase, fuel prices, transit fares, etc. on wellbeing. An interesting research 
direction with reference to QoL could be to investigate the effect on community 
wellbeing of supply-based factors like physical functions including road networks, 
cross-sectional design and vehicular traffic. Second, within literature on the built 
environment, studies on accessibility and connectivity suggest that high-density 
regions and proximity to public transport improve an individual’s QoL. However, 
the existing research lacks empirical evidence on how residential differences, 
spatial-temporal constraints or a lack of adequate connectivity influences individu- 
als’ QoL. 

Third, existing studies on transport externalities mostly focus on the effect of air 
pollution or traffic safety on health-related burdens, ignoring the socio-economic 
aspects. For instance, the impact of the socio-economic burden of transport related 
emissions and safety issues on QoL is still unknown. 

Fourth, regarding individual travel time use, travel needs are highly intercon- 
nected with work, family and social life. However, existing studies on QoL mostly 
focus on the health impacts of leisure and commuting trips. Daily travel activities 
also involve maintenance trips such as shopping and childcare (organized activities, 
help with homework, overseeing playtime and escorting), which involve car use and 
related stress. However, little is yet known about such negative impacts on health 
over time. Finally, travel attitudes and their association with QoL have mostly been 
analyzed in the context of travel satisfaction e.g., mode usage, travel time and their 
effect on hedonic wellbeing. However, travel motivations also involve emotion, atti- 
tudes and preferences that are bound by other long-term choices (destination, resi- 
dential occupation), as pointed out by De Vos et al. [68]. In this context, less is 
known in the literature about the impact of travel satisfaction on eudemonic wellbe- 
ing in the long run. 

Based on the key findings synthesized from the literature review, it can be seen 
that existing studies in transportation research have not completely addressed the 
QoL dimensions in transportation. Some conceptual models have been developed 
[16, 73-76]; however, these frameworks are firmly rooted in SWB understandings 
and focus on the mobility of older people, out-of-home or leisure-based activities, 
the built environment and travel choices. Overall, a complete picture of how trans- 
portation hinders or facilitates wellbeing is still missing from the literature, as trans- 
portation factors such as transport infrastructure and services, the built environment, 
transport externalities, travel time use and travel satisfaction and wellbeing dimen- 
sions such as social, community, economic, physical and psychological are interde- 
pendent and influence each other at various levels. This additionally calls for 
analysis of how each aspect of transportation impacts wellbeing not just at the indi- 
vidual level but also at the community level at large. 
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TRAWEL: A Conceptual Framework for Transportation 
Based QoL 


The study presented here proposes a conceptual framework (Fig. 24.1) that has been 
derived via an expert consensus process to holistically synthesize the impact of 
transportation on wellbeing at two levels. First, the elements of a transport system 
(infrastructure and services, built environment, and externalities) and its impact on 
wellbeing are discussed at the societal level (sections "Transport Infrastructure and 
Services (Mobility and Affordability)”, “Built Environment (Accessibility and 
Connectivity)" and “Transport Externalities (Safety and Air Quality)"). Second, 
individual travel needs and travel satisfaction with regard to travel elements and 
effects on their wellbeing are explained at the individual level (sections “Travel and 
Time Use (Travel Needs)" and “Travel Satisfaction (Travel Related Attitudes)"). 
The societal level is discussed first, as it is a context within which the individual 
behaviors occur. The conceptual framework TRAWEL will add to the existing lit- 
erature as it explains the association between five broader aspects of transportation, 
as derived from the literature review: (1) transport infrastructure and services, (2) 
the built environment, (3) transport externalities, (4) travel and time use and v) 
travel satisfaction and wellbeing measures. 


Transport Infrastructure and Services 
(Mobility and Affordability) 


Transport infrastructure and services form an integral part of the transport system as 
they enable people to cater for their travel needs. So far, as described in the state-of- 
the-art section above, the mobility dimensions of older people and the affordability 
of various pricing strategies have been discussed in assessments of QoL. Here we 
discuss how the underlying factors such as the road network and transport services 
affect individual QoL. 

The capacity of road infrastructure and the availability of transport services sup- 
port individuals' mobility and capacity to take part in a range of socio-economic 
activities. A recent study found a positive association between road and infrastruc- 
ture development and community satisfaction [77]. Meanwhile, the development of 
road infrastructure that cannot cope with increased mobility demands may lead to 
induced traffic [78, 79]. In such a case, the congestion is not appreciably reduced, 
but rather causes a strain on other mobility infrastructure services such as the avail- 
ability of public transportation, pedestrian facilities, Non-Motorized Traffic (NMT), 
or parking facilities. This is because cities follow car centric policies and most ser- 
vices are available far from residential neighbourhoods. For these reasons, public 
transport services become unavoidable for individuals with low car access. 

A recent paradigm shift in urban transport is the application of Intelligent 
Transport Systems (ITS) that cover a range of innovative demand-oriented services 
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in transportation. They include Mobility as a Service (MaaS), driver monitoring 
systems, vehicle technologies, automated driving, travel time prediction and con- 
gestion management. Each of these services aims to cut car dependency, curb traffic 
congestion, enhance autonomy, improve flexibility and safety, provide cost effective 
services and promote cleaner transport. Jones et al. [25] found positive attitudes 
towards technology interventions like e-bikes and QoL. 

Digital advancement in ITS plays a prominent role in shaping individual QoL, 
especially for transport disadvantaged groups who are economically and physically 
constrained. First, transition in existing mobility services comes at the price and 
disturbs individuals who are economically stretched. Evidence shows that there is a 
negative association between transport affordability for individuals and SWB [80, 
81]. This is because affordability for individuals largely depends on the resources 
they possess: physical capital (income), human capital (educational level) and social 
capital (participation in reciprocity networks), as pointed out by Hernandez [72]. 
Second, a study confirms that older people are less adaptable to ITS in their daily 
lives, contributing to unintended social exclusion [16]. 

Alternatively, the mobility options could be enhanced by on-demand mobility 
services, known as Flexible Transport System. Some flexible transport services 
such as shared taxicabs, shuttle vans, dial-a-ride services and para-transit services 
are offered as stand-alone services. These services generally cater for a specific 
group of the population or fill a specific need. Shared or collective flexible transport 
services include services such as ridesharing, car sharing and car-hailing services 
and bring together public and private transport services and volunteer transport ser- 
vices to provide cost-efficient connectivity for the entire community [82]. In this 
respect, the system supports the mobility needs of diverse groups such as women 
with young children, individuals living in rural communities, older people and dis- 
abled people. 

Besides a flexible transport service, well-kept pavements (sidewalks), lighting 
and crossings play a significant role in catering for essential mobility demands such 
as walkability and bike-ability. Both cycling and pedestrian infrastructure play a 
crucial role in enhancing the QoL, especially in the community where pedestrian 
mobility is predominant [83]. A study of older people revealed that problems with 
standing and a lack of walking infrastructure for at least 400 m decreases QoL [5, 
84]. Overall, the successful implementation of new mobility services depends on 
the pricing policy that constitutes the subsidies, differentiated pricing and different 
land use mixes. In this way, both mobility and affordability are ensured so that the 
transportation system enhances community wellbeing. 


Built Environment (Accessibility and Connectivity) 


The built environment encompasses a range of neighborhood services such as 
healthcare and education services and amenities such as green space, parks or shop- 
ping malls. So far, factors like urban density or proximity to public transport have 
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been predominantly discussed in the literature, ignoring the effect of constraints on 
QoL such as differences in residential location or a lack of connectivity. 

Access to services and amenities is ensured by the connectivity of road infra- 
structure and proximity of public transportation in neighborhoods. The ability to 
access essential services like health care facilities, grocery shops or public transit 
services enhances active travel behaviour (walking and biking), leads to significant 
health benefits [39, 85, 86]. Besides physical health, the built environment provides 
opportunities for social participation and a sense of belonging to the place in ques- 
tion. Studies find that proximity to public transport services and access by bicycle 
or on foot to green spaces enhance the possibility of social interactions among dif- 
ferent population groups, which positively influences QoL [87-92]. 

Built environments with well-connected street network accessibility and con- 
nectivity encourage individuals to take part in value-related out-of-home activities 
such as commerce, employment or recreation. These opportunities attribute a sense 
of belonging to a community and contribute to social wellbeing. However, if the 
built environment fails to provide neighbourhood accessibility, then it becomes a 
burden for the residents to travel and develop social connections. For instance, in 
residential suburbs on the outskirts of urban areas or in newly developed areas, the 
road network may lack continuity and provides limited access to services and ame- 
nities, which poses a threat to individuals with physical disabilities. Also, the eco- 
nomically constrained population live in the suburbs, while the high-income groups 
mostly live in the downtown region. Such spatial mismatches lead to unemployment 
as the skills possessed by the workers do not match the opportunities, as pointed out 
by Levinson and Krizek [93, p. 132]. It is proven that low-income and unemployed 
people mostly depend on public transport [94]. 

An absence of connectivity to public transit also reduces employment opportuni- 
ties. In such a situation, access to transit use may positively impact the QoL of low- 
income individuals, while the growth of car-reliant communities along the 
transit-oriented development (TOD) corridors negatively affects the QoL of low- 
income groups [35]. Individuals who have access to private cars can travel further 
in search of opportunities, which affects groups with no or limited car access. For 
instance, Cao et al. [95] found that living in suburban areas positively affects car 
dependence compared to living in urban areas. Altogether, a multidisciplinary 
approach to policy planning is warranted to move towards creating more activity- 
friendly communities and catering for individuals who are spatially or economically 
constrained. 


Transport Externalities (Safety and Air Quality) 


Transport externalities are the negative impacts of transport on the environment 
(e.g., congestion, air pollution, noise) and on society (e.g., road safety and public 
safety). So far, as described in section "Transportation Related QoL”, city level 
planning adopts three different strategies to address traffic externalities. First, road 
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safety is ensured through the physical attributes of road space (e.g., width and the 
cross-sectional attributes of pavements) and physical functions (e.g., zebra cross- 
ings, lighting), and by implementing road safety measures (e.g., creating awareness 
for road users, reducing drink driving and the strict enforcement of speed limits) 
[96]. Second, air quality is maintained by adopting vehicle emission control mea- 
sures, alternative fuel usage and enforcing regulations for older vehicles [97]. Third, 
congestion is addressed by reducing car usage through push-pull demand manage- 
ment measures such as parking, pricing and subsidies [98]. From the QoL perspec- 
tive, these strategies concern the physical and technical aspects of road infrastructure 
but ignore the social aspects. 

Studies across the global south (Latin America, Sub-Saharan Africa) and global 
north (Europe, United Kingdom and the United States) have documented how 
transport-related externalities such as congestion, vehicle emissions, noise pollution 
and road safety affect the livelihoods of various social groups [97, 99-108]. These 
studies together emphasize that externalities become a social problem when non- 
transport users are affected by such problems. Drawing on these studies, the following 
contributing factors are identified. First, individuals living in the downtown area or 
along the highways are predominantly exposed to greenhouse gas (GHG) emissions, 
which directly impacts their health and wellbeing. Second, individuals who reside in 
the outskirts with poor transportation suffer from congestion-related stress. Third, indi- 
viduals with a vulnerable mode of transport (walking or cycling) combined with poor 
quality transport are highly exposed to traffic accidents, potentially leading to perma- 
nent illness and injury. Fourth, deficient road infrastructure hampers the livelihood of 
residential communities as it cuts off the community, hampers social interaction and 
disturbs social wellbeing. Finally, the physical functioning of road infrastructure pose 
serious hazards to the social security of women, children and older people. 

Overall, it can be seen that transport externalities widely affect social groups and 
their wellbeing in the long run. To enable a healthy, safe, and communal way of life, 
it is recommended that policies addressing traffic externalities should consider 
social aspects such as income level differences, gender dimensions and age-related 
barriers besides the technical aspects of road infrastructure and vehicle design. 


Travel and Time Use (Travel Needs) 


Individuals’ daily travel needs are driven by their participation in out-of-home 
activities related to work, education, errands, sports, leisure and recreation. So far, 
as described in the section “Transportation Related QoL’, travel needs linked to 
leisure and commuting have been discussed. Such studies observe that travelling for 
recreational needs such as socializing with friends, visiting recreational places, 
sports and exercise involves physical movement and enhances enjoyment and 
amusement, which improves the physical and mental wellbeing of individuals [1, 
109—112]. Studies on job-related travel suggest that job or business travel causes 
commuting stress, starting from the pre-trip planning (work, family and travel 
arrangements), including the travel itself (delay and safety issues) and ending with 
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post-trip workloads (e.g., family commitments); however, such trips may have a 
positive impact on economic wellbeing such as income, productivity and personal 
development [113—119]. 

The following contributing factors have been identified in this context. First, the 
gender aspect recognizes that women carry a disproportionate share of unpaid work 
such as errands, shopping and childcare. These responsibilities leave them with no 
choice but to combine trips to perform work and family obligations. As suggested 
by many travel time-use studies within households, it is women who undertake 
shorter commutes, have less choice on the labour market, and have less access to a 
car in case of one-car households and traditional household patterns [120—123]. 
Second, research shows that individuals who travel with household members (e.g., 
joint travel for pickup and drop off, shopping activities, leisure trips) tend to travel 
more happily than when travelling alone [122, 124]. Third, studies on travel-based 
multitasking reportedly claim that women do more socializing and accompanying 
of children, while men do more work-related tasks [125, 126]. 

Aside from the gender differences and the travel time use perspective, overall, 
studies on wellbeing mostly focus on the professional sphere and recreational 
aspects but exclude the domestic sphere. However, such trips could affect daily 
moods and emotions. For instance, a recent study on travel time use finds that activ- 
ity during travel determines hedonic wellbeing or short-term happiness [127]. 
However, less is known in the literature about the impacts of unpaid trips, joint trips 
and travel-based multitasking on individual wellbeing in the long run. Henceforth 
QoL studies on travel behavior should jointly consider the various dimensions of 
travel needs at individual level and intrahousehold dynamics at household level, as 
pointed out by Sweet and Kanaroglou [128]. 


Travel Satisfaction (Travel Related Attitudes) 


Adopting an individual view on transportation and wellbeing also includes travel 
satisfaction. Studies have found a positive association between travel satisfaction 
and SWB (e.g., [5, 68, 75, 129-131]). SWB is commonly approached from two 
perspectives: travel hedonic or affect and travel eudemonia. So far, as described in 
section “Transportation Related QoL”, the association between travel satisfaction 
and hedonic wellbeing has been given primary importance. Some of the contribut- 
ing factors identified in the proposed framework are travel experience, mode use, 
residential and destination choice. 

Travel experience is characterized by hedonic feelings such as emotional state or 
mood and cognitive evaluations of an individual's context. Some common hedonic 
feelings include commuting stress in public transport, feelings of pleasure and hap- 
piness when riding a bicycle or driving a car. Such hedonic feelings influence travel 
decisions. For instance, individuals who enjoy cycling may reduce their car com- 
muting. In other cases, individuals who experience feelings of stress during conges- 
tion ought to seek alternate routes or modes. Ettema et al. [69] found that mode 
switching to sustainable modes triggers higher levels of SWB. 
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In the long term, hedonic wellbeing contributes to personal growth, finding pur- 
pose or meaning, and self-actualization, or achieving one’s full potential, termed 
eudemonic wellbeing [68]. Travel eudemonia relates to intrinsic motivations for 
traveling such as personal development (self-confidence, physical or mental health), 
competence (e.g., driving, riding), autonomy (freedom), and relatedness (e.g., social 
connections) [132]. For instance, commuting by car is connected to freedom, com- 
fort and self-identity, while riding a cycle may yield health benefits. 

Regarding mode use, numerous studies explore the association between active 
travel and hedonic-eudemonic wellbeing. For instance, Singleton [133] found that 
active transport such as walking and biking positively influences SWB [133]. In 
another study, Woodcock et al. [134] examined the impact of encouraging active 
transport on individuals’ health and found that active transport reduced heart dis- 
ease among men and reduced depression among women. In addition, a recent study 
found that active travel certainly enhances eudemonic wellbeing more than public 
transit [135]. Public transport users have needs and preferences, including reliabil- 
ity, convenience, safety, comfort, accessibility and affordability, that affect their sat- 
isfaction with the services provided. Some studies have shown that commuting by 
public transport reduces QoL compared to driving by car [136]. 

Regarding residential choice, people who live in their preferred neighborhood 
based on travel preferences (e.g., car lovers living in suburban types of neighbor- 
hoods) are more satisfied than people who do not. Participation in travel activities in 
larger cities is more weakly associated with life satisfaction than in smaller cities 
[1]. In addition, travel satisfaction might also affect performance of and satisfaction 
with activities at the trip destination. For instance, experiencing frequent positive 
emotions during travel may depend on the type of activity at the destination [74, 
137]. The studies addressing SWB have so far focused on traditional mobility 
options, as pointed out by Ettema et al. [69]. From a policy perspective, travel satis- 
faction with respect to new mobility options such as technology-based modes (elec- 
tric, autonomous) and shared mobility (car sharing, ride sharing, para transit) could 
provide further insights to understanding individual perceptions and preferences. 


Quantifying Transportation: Method and Measures 


In transport planning, a variety of methods have been used to identify potential 
indicators at micro and macro-level. Existing studies employ both qualitative and 
quantitative approaches to evaluate SWB in travel-based QoL. Building on the pre- 
vious literature, a series of indicators for each aspect of the conceptual framework 
are listed in Table 24.1. Discussion focuses on the various survey-based and auto- 
matic, sensor-based approaches to collecting data related to measures and recom- 
mended possible subgroups for the analysis; the aim is to understand group 
differences in the association between transportation aspects and wellbeing. 
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Transport Infrastructure and Services 


The data for transport infrastructure and services can be obtained using five mea- 
sures: (1) the physical attributes of the transport network such as road capacity, road 
geographical features and traffic composition can be collected using the Geographical 
Information System (GIS) data model, for example managed by the local city or 
regional authorities or other stakeholders, as pointed out by Croce et al. [138]; 
mobility-relevant data such as frequency of trips, travel distance and duration [139]; 
(2) information about individuals’ use of different travel modes such as flexible 
transport services, ride sharing services, paratransit, car availability, walking and 
cycling may be obtained from national or regional household travel surveys, as in 
the case of most recent studies [140, 141]; (3) data on public transportation services 
such as level of service, mobility and accessibility factors can be collected using 
GIS tools complemented by vehicle location system and smart card fare data [142, 
143]; (4) automated data collection systems in ITS can provide data on automated 
fare collection, automatic passenger counts including information about passengers 
and vehicle spatial attributes such as the GPS positioning of vehicles, time of the 
day, inbound/outbound details of actual arrival and departure times at the beginning 
of the route, end of the route and at every bus stop along the route [144]; and (5) data 
attributed to transport costs (taxes, accident costs, medical expenditure) can be 
obtained from respective transit agencies, insurance agencies and medical care 
facilities. 


Built Environment 


The geospatial method can be adopted to examine the association between an indi- 
vidual’s participation in an activity (e.g., going to a library, going to a supermarket, 
etc.) and geographic accessibility [86]. Available land-use and land-cover data from 
local city or regional authorities or other stakeholders and from open sources like 
European Union open data portals can be used to understand the spatial distribution 
of different land uses. Travel survey data of residents living in traditional neighbor- 
hoods, suburbia [95] and other regions can be used to compare different neighbor- 
hoods’ mobility characteristics. Both the geospatial data and travel survey data can 
also be obtained from secondary sources like government organizations and certi- 
fied private agencies. The data attributed to the transport network must include 
walking trails and bicycle networks [145] and the traditional road network data. 
Extending these data sources, geospatial techniques like those leveraged by D’ Orso 
and Migliore [83] could be used to assess walkability to and from transport services 
based on practicability, pleasantness and safety. In addition to these data techniques, 
transit network data could be collected from local transport agencies to analyze the 
connectivity and proximity of transit services [91]. 
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Transport Externalities 


The data for traffic congestion such as traffic volume, mode classification and level 
of service can be obtained from household travel surveys and traffic count data 
[146]. In addition, data on safety in daily travel can be obtained through face-to-face 
interviews and online feedback forms or can be incorporated as a component in 
household survey data. Air quality data from air quality monitoring stations can 
help model exposure level to gaseous pollutants and particulate matter. Recent 
advances in technology interventions have led to portable instruments for measur- 
ing vehicular emissions on-board [147, 148] that could also be employed in data 
collection. As an individual’s exposure level also varies with different mobility pat- 
terns, it can be measured using the Portable Air Quality Monitoring sensors (PAMs) 
over a period [149]. These data can form a basis for modelling the health-related 
QoL of the public and vulnerable groups. Additionally, a noise emission model can 
capture the noise generated by the impact of road traffic on vehicle kinematics [100]. 

To measure the social impact of transport externalities on social groups, data on 
individual attributes (gender, age, employment status and personal income, etc.) and 
household attributes (household size, number of children and dwelling type, etc.) 
are required. These data are mostly derived from structured self-reports (e.g., house- 
hold travel surveys) where the respondents (mainly the household head) self-report 
individual, household and travel attributes. Health data can be collected through 
national health surveys [150]. 


Travel and Time Use 


Travel and time use related data can be obtained from the cross-sectional time use 
surveys. A typical time-use survey comprises 24 h activity patterns of each respon- 
dent for three random, representative days. The data derived from these surveys (1) 
includes in-home and out-of-home activities classified in four broader groups: each 
activity (paid work, unpaid work, leisure) and associated travel; (2) includes pri- 
mary and secondary activities that allow analysis of travel-based multitasking; and 
(3) provides details about whether the individual performs the activity alone or with 
other household members, which could be used for joint household travel analysis. 
Besides time use surveys, the National Survey of Parents, the Family study includ- 
ing surveys and the Experience Sampling Method can provide information about 
primary and secondary activities [151]. Time spent on activities could additionally 
be obtained from self-reported longitudinal or cross-sectional epidemiological 
investigations conducted either online or using face-to-face surveys. Also, travel 
diary applications can be utilized to obtain the mobility patterns adopted for differ- 
ent in-home and out-of-home activities. In addition, health behavior associated with 
active travel can be obtained using the Harmonised European Time use Survey [152]. 
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Travel Satisfaction 


The travel-related attitudes on various scales such as satisfaction, liking, happiness, 
enjoyment and subjective valuation of time over travel modes, destination and activity 
can be qualitatively derived from virtual passive data (including location data) and 
from smartphone applications (mobile phone-based surveys and locations captured 
via WiFi, GPS or GSM data). For instance, Susilo and Liotopoulos [153] used various 
data collection formats, including an online web-based, real time questionnaire using 
an Android application, and a focus group interviewing method to collect data related 
to travel satisfaction. In another study, Kwon and Lee [154] adopted a qualitative and 
quantitative approach to measure the travel satisfaction. Quantitative data is collected 
using face-to-face interviews, semi-structured interviews and narrative interviews, 
and qualitative data is derived from longitudinal measurements with 15-day intervals 
before and after a given trip, leveraging the qualitative dataset collection via an online 
portal. The data on individual satisfaction about mode usage reflect attitudes towards 
traffic congestion, road conditions and availability of alternate or public transport 
services. Such information may be collected by using face-to-face interview surveys 
and questionnaires [17], leveraging online social platforms [155], analyzing citizen 
satisfaction data from annual surveys [156], or administering perceived quality sur- 
veys [157]. Apart from the readily available data, perceived benefits and burdens can 
be acquired via administration of validated QoL questionnaires. 

Residential choices can be determined by subjective measures, such as satisfac- 
tion with neighbourhood attractiveness, comfort, convenience and safety. These 
data can be obtained from perception and attitudinal surveys administered in face- 
to-face interviews, via mobile apps or online forms. Factors contributing to eude- 
monic wellbeing [132] can be modelled across different socioeconomic and 
sociodemographic groups. For instance, the overall physical health of participants 
and the daily frequency of feeling rushed could be collected using self-reported 
questionnaires [158]. Example studies in this area incorporate physical and mental 
health data using National Household Health Surveys [159] or World Health Surveys 
[160, 161]. In addition, the Questionnaire for Eudemonic Wellbeing (QEWB) and 
the Flourishing Scale (FS) [162] can be leveraged. 


Discussion 


In this chapter we discuss transportation aspects such as transport infrastructure and 
services, the built environment, transport externalities, travel time use and travel 
satisfaction, and familiar wellbeing dimensions such as social, community, eco- 
nomic, physical and psychological, which are interdependent and influence each 
other at various levels. Overall, the transportation aspects affect the wellbeing of 
diverse social groups in the following ways. 

First, poor transport infrastructure and limited mobility services affect mobility 
and increase transportation costs. This widely affects the community wellbeing of 
social groups, especially impacting groups with low car access, the older population 
and poor income groups. Second, built environments that lack local and regional 
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accessibility, transport connectivity, green or public spaces and are subject to discon- 
nected services disrupt social integration and isolate the community. As the social 
support of the community is threatened, the social wellbeing of vulnerable groups is 
affected. This further influences social conditions and potentially leads to social 
problems such as unemployment and crime. Third, transport-related air and noise 
pollution negatively impacts the health of residents who are constrained by income, 
mode and residential location, while lack of traffic safety leads to road accidents and 
affects the overall wellbeing (economic, social, mental and physical health) of resi- 
dents. Fourth, feelings of being rushed during paid work trips, additional trips to 
cater for unpaid work demands such as errands, escort and shopping, and the lack of 
(pure) leisure related trips jointly affect the mental and physical wellbeing of indi- 
viduals, in particular working women with traditional gender role orientations. 
Finally, low travel satisfaction due to commuting stress, traffic congestion or delays, 
a low quality of public transport services, and a lack of interest in cycling or walking 
affect the physical health and psychological wellbeing of individuals. 

The functions of the transportation aspects (transport infrastructure and services, 
built environment, transport externalities, travel time use and travel satisfaction) 
overlap one another, while wellbeing, on the other hand, is contextual and multifac- 
eted and varies according to social groups. For instance, if transportation services 
are not available to all segments of the population equitably, then the problem of 
social injustice emerges in a society where the vulnerable groups who contribute 
less to congestion and pollution are those that are most exposed to these problems. 

At the societal level, transport policy planning requires the consideration of hori- 
zontal equity and vertical equity in transportation, as suggested by many studies 
[163-166]. Horizontal equity enables the equal distribution of transport services 
among groups with the same transport needs, while vertical equity accounts for 
social differences between groups with different transport needs. For instance, on 
the horizontal level, the availability and proximity of public transport services 
improve the transport mobility of all residents in the community, while on the verti- 
cal level, reasonable ticket fares in public transport services enhance affordability 
for low-income groups, seat availability improves accessibility for individuals with 
physical constraints (wheelchair users, those with impaired sight or older people) 
and safety enhances women's mobility. 

At the individual level, travel demand measures should consider the intrinsic 
motivations of individuals to travel. For instance, the first-and-last mile (F&LM) 
problem and the lack of proximity to services (employment, health and education) 
may push individuals to increase car trips and long-distance travel. In such cases, 
transport pricing policies are crucial as they significantly impact travel choices and 
help in regulating vehicle traffic and managing infrastructure and natural resources 
(e.g., vehicle taxation in France, Germany and Sweden) [167]. Alternatively, land 
use policies like transit-oriented development can enhance life satisfaction by 
enabling ease of access to essential services and promoting safe out-of-home activi- 
ties. Likewise, strategies for integrating multi-modal trips provide better connectiv- 
ity and improve individuals’ wellbeing. In addition, policy experimentation to 
enhance subjective measures (comfort, convenience, safety, reduced stress levels) 
can encourage usage of public transport. For example, the tele-bus on-demand 
transport service in Australia and Germany [168, 169] aims to provide a bus service 
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on demand in less populated areas; in this way both individual wellbeing and com- 
munity wellbeing can be improved. 

The study has limitations. First, both wellbeing and transportation are multifac- 
eted and complex, as well as unfolding and evolving over time, and this discussion 
addresses only part of this complex relationship, captured momentarily. Second, the 
internal and the external validity of the wellbeing measures extracted from the con- 
ceptual framework with respect to travel context remain unknown. Third, although 
the study conceptually untangles the well-being and travel attributes, the question of 
a suitable methodological approach and deployment of statistical methods remains. 
Further, adopting QoL as the primary goal in assessing transport systems and under- 
standing travel behaviors requires more research on the framework, measures and 
methods, and clear demonstration with case studies and empirical evidence. Finally, 
the study does not address the effect of recent technological changes in transporta- 
tion that aim to improve individual QoL; the study focuses on the assessment aspects 
of transportation's contribution to individual QoL [170]. Some of the additional 
technological innovations not considered here include the Quantified Self (QS) 
technologies including wearables to self-track individual behavioral patterns that 
may influence their behavioral choices and lifestyles [171, 172]; smart city innova- 
tions e.g., automated/electric vehicles, or advanced traffic management systems and 
urban mobility apps that aim to improve transport mobility [173]. Future research 
could extend the current study to examine whether these technological innovations 
in transportation improve wellbeing. 


Conclusive Remarks 


This chapter puts forward and discusses the hypothesis that transport systems affect 
wellbeing on the societal level, while travel behavior influences wellbeing on the 
individual level. The extensive literature method identifies the relevant transporta- 
tion aspects that impact wellbeing at the individual and societal levels. Based on the 
extant literature, the conceptual framework TRAWEL is put forward to understand 
the association between transportation aspects and wellbeing. The determinants of 
the transport system such as the mobility, affordability, accessibility, connectivity, 
safety and air quality influence community or social wellbeing, while travel needs 
and travel attitudes influence the psychological and physical wellbeing of individu- 
als. Such interconnectedness between transportation aspects and wellbeing has not 
been fully explored elsewhere in transport research to date. 

Following the conceptual framework TRAWEL, the review of existing measures 
and survey procedures suggests the scope for quantifying the impact of transportation 
aspects on QoL. Finally, the discussion on policy relevance at the societal level sug- 
gests an agenda of addressing horizontal and vertical equity in transport system plan- 
ning, to enable community, social and economic wellbeing. Additionally, it suggests 
the need for further understanding of the intrinsic motivations of travel needs and 
travel satisfaction with transport at the individual level, which may be leveraged to 
enhance individuals’ physical and mental wellbeing. To advance understanding of the 
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impact of transportation on individual wellbeing, future research should broaden the 
intersectionality between transportation aspects and wellbeing dimensions. 
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Daily behaviors influence an individual's health and overall quality of life (QoL). 
Specific patterns of behavior such as smoking, having a poor diet, being physically 
inactive, or consuming alcohol influence the long-term development of chronic dis- 
eases such as type 2 diabetes, chronic obstructive pulmonary disease, or cardiovas- 
cular disease [1]. In the US alone, chronic diseases are becoming increasingly 
common, and about half of all deaths can be attributed to preventable behaviors and 
exposures [2, 3]. Figure 25.1 represents the proportional contribution of various fac- 
tors to causes of death, including genetic, behavioral, and systemic aspects of one's 
life [2, 3]. 

The influence of daily behavior on the development of chronic diseases and, 
ultimately, mortality is a stark example of how behaviors influence QoL in the long 
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Fig. 25.1 Proportional contribution of various factors to mortality 
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term. It is therefore important for health practitioners and researchers, as well indi- 
viduals themselves to accurately to accurately quantify individuals’ behaviors, and 
psychological states to improve their awareness of these factors, encourage neces- 
sary habit changes, and provide more individualized treatment approaches. 
Currently, individuals’ behaviors and psychological states are mainly quantified via 
methods such as the use of self-reported measures, many of which consist of vali- 
dated scales referred to as patient-reported outcomes (PROs) [4]. Among other 
behaviors, PROs may include self-reported measures of physical activity (e.g., IPAQ 
[5]), sleep (e.g., PSQI [6]), or nutritional habits (e.g., Mediterranean diet score [7]). 

To characterize the current landscape of tools and methods used to quantify QoL, 
the chapters of this book discuss various PROs that have been used to assess behav- 
iors relating to all 24 aspects of an individual’s QoL as defined by the World Health 
Organization [8]. In particular, each chapter identifies methods and instruments for 
assessing a particular variable of QoL; some of the methods and instruments are 
considered gold standards in the field of QoL assessment. A summary of the exist- 
ing self-reported QoL measures that are most commonly used in adult populations 
can also be found on the website of the QoL lab, pros.gol-unige.ch, which will be 
regularly updated in the future. 

As is discussed extensively throughout the book, these self-reported PROs have 
multiple limitations, including the infrequency with which they are collected, the 
subjective nature of responses, their reliance on memory recall, and their suscepti- 
bility to the influence of social norms. Additionally, self-reported PROs are typi- 
cally performed outside of the context in which the assessed behavior occurs. 
Therefore, the use of personalized and miniaturized technological innovations, 
including smartphones [9], mobile applications [10], and wearables [11, 12], is pro- 
posed in this book as an alternative means of quantifying QoL that can enable the 
continuous and more accurate assessment of daily life behaviors that contribute to 
or result from an individual's QoL. 

The advantages of using these technologies include the fact that they allow more 
frequent measurements than PROs, provide longitudinal data, are objective, sensory- 
based, and non-judgmental, and permit a context-rich approach to the assessment of 
individual states and behaviors [13]. The behavioral and health outcomes captured 
by these personal technologies, the US Food and Drug Administration (FDA) calls 
“digital health technology tools (DHTTs) [14, 15], are referred to as technology- 
reported outcomes (TechROs) [4]. According to regulatory bodies such as the FDA, 
TechRO data is an example of real-world data (RWD). The RWD includes elec- 
tronic health records, information from insurance claims and billing, registries, and 
data collected from medical devices used outside of clinical settings. The analysis 
of RWD produces a broad range of objective evidence that can be used for research 
and regulatory purposes, including information about individuals' behavior and the 
potential advantages or risks of specific behavioral or pharmacological interven- 
tions [15, 16]. The emergence of standardized definitions of TechROs and the grow- 
ing use of TechROs as a data source are important to note in the context of the 
present book. The work of the QoL lab defined the larger field of “quality of life 
technologies" (QoLTs) to which TechROs belong. These technologies encompass a 
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broad range of tools that can be leveraged to assess, maintain, improve, prevent 
decline in, or compensate for one’s life quality, which can be applied at the indi- 
vidual, interpersonal, community, group, or population level [17]. 

Expanding on the research on QoLTs, the contributors to this book discuss and 
evaluate various TechROs that have been used to assess behaviors relating to differ- 
ent aspects of an individual’s QoL, as well as technologies that could be applied to 
the collection of TechRO data in the future. Moreover, each chapter elaborates on the 
role of the emerging quantified self (QS) movement in the context of the QoL vari- 
able it focuses on. The QS movement! involves a group of highly motivated individu- 
als who leverage personalized technologies to assess and intervene in their behaviors 
with the ultimate goal of realize the QS ideal of “knowing thyself.’ Engaging in the 
QS practices presented in this book may not be possible for the average individual 
today. However, the book's discussions highlight practices that may be common- 
place in the future, as they demonstrate the feasibility of various technology-enabled 
approaches to behavior self-management in particular areas of one's daily life. 

Before TechROs can be leveraged in clinical practice, many challenges need to 
be overcome. These include ensuring the accurate collection of technology-based 
data [18], interpreting the data in its appropriate context, and resolving potential 
ethical dilemmas in the ways data is collected and used [19, 20]. Ultimately, the data 
collected through use of a given technology must accurately represent its defined 
variable, and the individual must accept the use of these technologies in their daily 
life for their relevant purposes [21 ]. 

In spite of these, however, the book overall presents a positive outlook concern- 
ing the state of the art and current developments in the use of TechROs and the future 
potential for leveraging TechROs to supplement PROs in the assessment of individu- 
als’ behaviors and health and life quality outcomes. In one such scenario, individu- 
als may be assessed using TechROs throughout their lifetime, with the results of 
assessments being leveraged to improve their health and levels of care, including 
self-care. Figure 25.2 presents a vision of the future quantification of individuals’ 
behavioral data. Drawing upon comparisons to the genome (a term that denotes the 
entirety of an organism's genetic material) and its common application in current 
genetics testing, the collection of the behavioral data is called one's “behaviome.” 
Ultimately, the mapping of this behaviome through behavioral assessments could 
improve diagnoses, treatments, and practices for the prevention of diseases and, if 
applied regularly over an individual's lifetime, improve their overall health outcomes. 

The technologies used to collect TechRO data can empower individuals to 
become “co-producers” of and experts in their own health and QoL, both in the 
short and the long term [22]. Medicine may thereby be transformed into a more 
personalized, predictive, participatory, and preventative system. Meanwhile, quan- 
tifying one’s QoL might become a new norm for individuals and populations at 
large, enabling the transition from healthcare to self-care that the world needs. 


! QuantifiedSelf, quantifiedself.com, visited May 23, 2021. 
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Fig. 25.2 Quality of life technologies: from genome to behaviome 
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